SCMC in Frequency-Aware Audio-Visual Segmentation

• SCMC is a frequency-aware module that enforces semantic consistency between decomposed audio and visual features, addressing modality-specific frequency challenges.
• It employs a mixture-of-experts architecture with bidirectional cross-modal attention, dynamically selecting experts based on global audio-visual cues.
• SCMC significantly boosts segmentation performance in complex, multi-source scenarios when integrated into the FAVS pipeline and validated on benchmarks.

Searching arXiv for the specified SCMC paper and closely related cross-modal consistency work. Synergistic Cross-Modal Consistency (SCMC) is a module introduced within the Frequency-Aware Audio-Visual Segmentation (FAVS) framework for audio-visual segmentation. In that formulation, SCMC is designed to enforce semantic consistency across decomposed audio-visual frequency bands and to preserve modality-specific features, operating after the Frequency-Domain Enhanced Decomposer (FDED). It receives frequency-decomposed and reconstructed feature maps for video $(\hat{F}_{output}^v)$ and audio $(\hat{F}_{output}^a)$, uses a mixture-of-experts architecture with attention-based experts, and produces aggregated, semantically-enriched, and cross-modal-aligned features $(\tilde{F}_v, \tilde{F}_a)$ for subsequent transformer decoding and segmentation (Shen et al., 23 Sep 2025).

1. Concept and scope

In the FAVS framework, SCMC addresses a specific problem: recent audio-visual segmentation methods overlook the inherent frequency-domain contradictions between audio and visual modalities, namely the pervasively interfering noise in audio high-frequency signals versus the structurally rich details in visual high-frequency signals. FDED performs residual-based iterative frequency decomposition to discriminate modality-specific semantics and structural features, while SCMC performs the complementary stage of recomposition by leveraging a mixture-of-experts architecture to reinforce semantic consistency and modality-specific feature preservation through dynamic expert routing (Shen et al., 23 Sep 2025).

The module is therefore not a generic fusion block. Its stated role is to recompose modality-specific features by enforcing cross-modal consistency with frequency awareness, leveraging complementary cues and restoring features lost to frequency band suppression during FDED. The paper characterizes the overall two-stage design as discriminative decomposition followed by synergistic recomposition, with SCMC being the mechanism responsible for the latter (Shen et al., 23 Sep 2025).

A plausible implication is that SCMC should be read as a frequency-aware consistency mechanism rather than as a modality-agnostic consistency regularizer. In the source formulation, its definition is inseparable from the decomposition outputs of FDED and from the downstream segmentation decoder.

2. Architectural composition

SCMC receives frequency-decomposed and reconstructed feature maps for video and audio from FDED. It employs $N_e$ experts, each expert being an attention module specializing in different frequency-band-specific representations. Each expert includes a Spatial-Temporal-Channel (STC) feature enhancer and Bidirectional Cross-Modal Attention (BCA). STC enhances features by modeling spatial, temporal, and channel cues relevant to frequency bands, while BCA performs bidirectional attention in both directions, audio $\rightarrow$ video and video $\rightarrow$ audio, producing expert outputs $(\bar{F}_v^e, \bar{F}_a^e)$ (Shen et al., 23 Sep 2025).

The module’s cross-modal consistency is achieved through several coordinated design elements. First, the mixture-of-experts structure allows different experts to specialize on different frequency band correlations. Second, BCA ensures that features from one modality attend to and integrate information from the other. Third, dynamic routing weights for visual features are generated by analyzing global audio features, and vice versa, so that one modality guides expert selection in the other. The paper explicitly describes this as complementary guidance via dynamic routing (Shen et al., 23 Sep 2025).

This architecture is also intended to restore suppressed features. The stated example is that visual cues can guide audio experts to focus on relevant sound source cues even if high-frequency audio was suppressed in FDED. In that sense, SCMC preserves modality-specific features not by isolating the modalities, but by selecting and recombining them under cross-modal guidance.

3. Formal mechanism

For each expert $e$, the bidirectional cross-modal attention operations are written as

$$\bar{F}_v^e = f_{a \rightarrow v}^e(\text{STC}_q^e(\hat{F}_{output}^v),\, \text{STC}_k^e(\hat{F}_{output}^a),\, \text{STC}_v^e(\hat{F}_{output}^a))$$

and

$$\bar{F}_a^e = f_{v \rightarrow a}^e(\text{STC}_q^e(\hat{F}_{output}^a),\, \text{STC}_k^e(\hat{F}_{output}^v),\, \text{STC}_v^e(\hat{F}_{output}^v)).$$

These equations define the expert-specific cross-modal transformations after STC enhancement (Shen et al., 23 Sep 2025).

The routing weights are produced from the other modality: $(\hat{F}_{output}^a)$0 The paper states that this encourages the model to use information from one modality to select expert contributions in the other, reinforcing alignment and complementarity (Shen et al., 23 Sep 2025).

SCMC further uses entropy-controlled dynamic top-$(\hat{F}_{output}^a)$1 selection: $(\hat{F}_{output}^a)$2

$(\hat{F}_{output}^a)$3

The system activates the top-$(\hat{F}_{output}^a)$4 experts for each sample, normalizes their weights $(\hat{F}_{output}^a)$5, and aggregates the outputs: $(\hat{F}_{output}^a)$6 The paper describes this dynamic routing as inspired by StableMoE and extended with entropy-based adaptive selection, with the stated advantages of expert diversity, avoiding collapse to a subset, and enabling sample-wise specialization (Shen et al., 23 Sep 2025).

4. Position within frequency-aware audio-visual segmentation

Within FAVS, the processing order is explicit: audio and visual inputs are encoded and passed through FDED, which uses frequency decomposition to separate, enhance, or suppress frequency bands, such as edge detail for vision and noise suppression for audio. The resulting outputs are then fed to SCMC. SCMC is said to recompose the modality-specific features by enforcing cross-modal consistency with frequency awareness, thereby mitigating information loss or distortion incurred during frequency band suppression (Shen et al., 23 Sep 2025).

The paper frames the interaction between FDED and SCMC as synergy. FDED supplies discriminative decomposition; SCMC supplies synergistic recomposition. The combined effect is described as enabling robust, semantically consistent, and precise audio-visual segmentation, particularly in challenging multi-source or noisy scenarios. The largest gains are reported in multi-source and complex acoustic-visual scenes, especially on MS3 and AVSS (Shen et al., 23 Sep 2025).

A common misunderstanding would be to treat SCMC as a stand-alone supervision term. The source paper states the opposite: SCMC itself does not introduce a unique loss; supervision is provided at the segmentation level via the standard mask-based segmentation loss functions used in the overall transformer decoder at the end of the FAVS pipeline. SCMC is trained end-to-end with the rest of the network, and the adaptive routing and expert specializations emerge via the gradients from the segmentation loss (Shen et al., 23 Sep 2025).

5. Empirical evidence

The ablation results reported for S4 and MS3 isolate the contribution of SCMC. The baseline obtains $(\hat{F}_{output}^a)$7 and $(\hat{F}_{output}^a)$8 on S4 $(\hat{F}_{output}^a)$9 and $(\tilde{F}_v, \tilde{F}_a)$0, and $(\tilde{F}_v, \tilde{F}_a)$1 and $(\tilde{F}_v, \tilde{F}_a)$2 on MS3 $(\tilde{F}_v, \tilde{F}_a)$3 and $(\tilde{F}_v, \tilde{F}_a)$4. Adding FDED yields gains of $(\tilde{F}_v, \tilde{F}_a)$5, $(\tilde{F}_v, \tilde{F}_a)$6, $(\tilde{F}_v, \tilde{F}_a)$7, and $(\tilde{F}_v, \tilde{F}_a)$8, respectively. Adding SCMC on top of FDED yields $(\tilde{F}_v, \tilde{F}_a)$9 and $N_e$0 on S4 $N_e$1 and $N_e$2, and $N_e$3 and $N_e$4 on MS3 $N_e$5 and $N_e$6. The paper explicitly characterizes the SCMC increment as a significant performance boost, especially notable in complex, multi-source scenes (Shen et al., 23 Sep 2025).

The full FAVS framework with SCMC is reported to achieve state-of-the-art results on S4, MS3, and AVSS benchmarks. An ablation on the number of experts indicates that performance peaks at 4 experts, with further increase leading to diminished or redundant returns, which the paper interprets as confirming the specialized yet diverse nature of expert learning (Shen et al., 23 Sep 2025).

The qualitative evidence is also cross-modal in emphasis. Grad-CAM visualizations are reported to show that, before SCMC, audio and visual feature maps exhibit spatial inconsistency, whereas after SCMC they show improved spatial overlap and mutual focus on the relevant objects. Visualizations of individual expert feature maps and t-SNE plots of expert outputs are reported to show that each expert learns unique, complementary features and that expert outputs cluster distinctly, indicating non-redundant, specialized learning across the frequency spectrum (Shen et al., 23 Sep 2025).

6. Relation to the broader literature on cross-modal consistency

SCMC belongs to a broader family of consistency-oriented multimodal methods, but its mechanism is specific. In vision-language retrieval, CoVLR coordinates the cross-modal consistency objective and the intra-modal relation preserving objective through a meta-optimization based strategy, with the two objectives acting as meta-train and meta-test tasks (Yang et al., 2023). In cross-modal matching under noisy correspondence, BiCro operationalizes bidirectional cross-modal similarity consistency by estimating soft correspondence labels from bidirectional similarity relationships to clean anchors (Yang et al., 2023). These approaches share the language of consistency, but they target retrieval and noisy matching rather than frequency-aware audio-visual segmentation.

In audio-visual speech separation, VisualVoice enforces cross-modal consistency between facial appearance, lip movements, and audio signals by jointly learning audio-visual speech separation and cross-modal speaker embeddings, with explicit cross-modal matching and speaker consistency losses (Gao et al., 2021). In multimodal spiking neural networks, S-CMRL uses semantic alignment optimization to align cross-modal features within a shared semantic space while retaining a cross-modal residual learning structure (He et al., 18 Feb 2025). In isolated sign language recognition, CCL-SLR aligns RGB and pose feature spaces through single-modal and cross-modal contrastive learning, with Motion-Preserving Masking and Semantic Positive Mining used to improve cross-modal consistency from the perspectives of data augmentation and sample similarity (Wu et al., 16 Mar 2025).

The evaluation literature also sharpens the meaning of consistency and synergy. XModBench measures cross-modal consistency in omni-modal LLMs through task competence, modality disparity, and directional imbalance, and reports that current models remain far from truly modality-invariant reasoning (Wang et al., 16 Oct 2025). A separate explainability benchmark introduces Synergistic Faithfulness $N_e$7, a metric rooted in the Shapley Interaction Index that isolates the joint Harsanyi dividend between modalities, thereby distinguishing cross-modal synergy from mere redundancy (Ky et al., 21 May 2026). This suggests that SCMC, in the FAVS sense, should be distinguished from weaker notions of multimodal compatibility: the module is intended not only to align modalities, but also to preserve modality-specific features while recomposing them under sample-specific expert selection.

Across these strands, the recurring theme is that effective multimodal learning depends on more than simple feature fusion. In the specific sense defined by FAVS, SCMC denotes an adaptive MoE attention module that bridges decomposed audio and visual features in the frequency domain and is empirically associated with improved robustness and accuracy in audio-visual segmentation (Shen et al., 23 Sep 2025).