TAViS: Audio-Visual Segmentation Framework

• TAViS is a cross-modal framework for Audio-Visual Segmentation that integrates text bridging to align audio, image, and text modalities for pixel-wise segmentation.
• It couples ImageBind and SAM2 through ImageBind-guided Query Decomposition and hybrid prompting, enabling effective decomposition of audio queries into object-level representations.
• Empirical results show TAViS achieves superior performance across single-source, multi-source, and zero-shot settings with precise, memory-consistent segmentation outcomes.

TAViS is a framework for Audio-Visual Segmentation (AVS) that addresses the cross-modal alignment problem by coupling a multimodal foundation model, ImageBind, with a segmentation foundation model, SAM2, through a text-bridged design (Luo et al., 13 Jun 2025). In this setting, AVS aims to generate pixel-wise segmentation maps of sounding objects in a scene, and the central difficulty is learning object-level correspondences between audio events and their visual counterparts under large intra-class diversity, including timbre, background context, and appearance variations. TAViS is defined by three core elements: ImageBind-guided Query Decomposition (IBQD), a text-bridged hybrid prompting mechanism, and text-bridged alignment supervision. The framework is reported to achieve superior performance on single-source, multi-source, semantic datasets, and to excel in zero-shot settings.

1. Problem setting and design rationale

Audio-Visual Segmentation seeks pixel-wise masks for sounding objects. The formulation emphasized in TAViS is not merely multimodal fusion, but cross-modal alignment at object level. The cited work identifies a fundamental limitation of prior AVS systems: many methods rely on single-modality knowledge, such as a visual foundation model like SAM or Semantic-SAM, or an audio model such as BEATs, while others combine models such as SAM and ImageBind in an off-the-shelf, decoupled manner. In both cases, the modality gap is left largely unresolved, and interaction between the multimodal and segmentation components is insufficient for effective knowledge transfer (Luo et al., 13 Jun 2025).

The proposed remedy is to couple ImageBind, which provides aligned audio, image, and text encoders, with SAM2, which provides precise region segmentation and temporal consistency. The difficulty is that SAM2 and ImageBind reside in different feature spaces, and supervision from segmentation loss alone is insufficient. TAViS therefore introduces text as a bridge between modalities and between model families.

This design suggests that the framework treats text less as a final prediction target than as a semantic anchor. In the reported formulation, text prototypes reduce intra-class noise by anchoring both audio and image to compact textual concepts, while preserving modality-specific information needed for localization and decoding.

2. System architecture

The architecture combines SAM2 and ImageBind in a coupled pipeline (Luo et al., 13 Jun 2025). SAM2 contributes an image encoder $F_E$, memory attention $F_M$, and mask decoder $F_D$. ImageBind contributes encoders $E_A$ for audio, $E_I$ for image, and $E_T$ for text, each producing trunk features and a cls token.

The visual input is a frame sequence $I \in \mathbb{R}^{T\times H\times W\times C}$, passed through SAM2’s image encoder $F_E$, augmented with audio-guided adapters, to yield multi-stage image features for $F_D$, with temporal memory in $F_M$ when needed. The audio waveform $F_M$0 is encoded by $F_M$1 into an audio trunk feature $F_M$2 and an audio cls token $F_M$3. The text branch uses class labels formatted as "A [cls]", encoded by $F_M$4 to create text prototypes. Pseudo-text is synthesized from audio queries and is used both for prompting and for supervision.

The interface from ImageBind to SAM2 is prompt-based. The sparse prompt $F_M$5 combines pseudo-text embeddings and audio-specific embeddings. The dense prompt $F_M$6 is derived from ImageBind’s image cls token, tiled and added to every SAM2 image embedding location. The former injects class-prototype semantics plus instance-level audio detail; the latter injects global visual context aligned with ImageBind’s shared space.

Trainable and frozen components are explicitly separated. ImageBind encoders $F_M$7, $F_M$8, and $F_M$9 are frozen. The SAM2 image encoder is frozen, while memory attention $F_D$0 and mask decoder $F_D$1 are fine-tuned. IBQD, prompt MLPs, audio-guided adapters, and the necessary projection and gating modules for bridging are trained as part of the AVS model.

3. ImageBind-guided Query Decomposition

IBQD is introduced because SAM2 expects object-level queries, whereas input audio may contain multiple mixed sound sources (Luo et al., 13 Jun 2025). A naïve learnable decomposition would discard the aligned ImageBind space, so TAViS instead decomposes audio into object-level queries while preserving ImageBind’s alignment properties.

Given the audio trunk feature $F_D$2 and audio cls token $F_D$3, the method introduces learnable queries $F_D$4 and applies multi-head cross-attention twice. First, the learnable queries attend to the audio trunk:

$F_D$5

Second, object-specific biases are created through a linear layer, added to $F_D$6, and re-attended to the audio trunk:

$F_D$7

The output is a set of decomposed queries $F_D$8 aligned with individual sources. The reported interpretation is that IBQD preserves ImageBind-aligned audio space during object-level decomposition, thereby improving query quality for SAM2.

Ablation results attribute a substantial fraction of the framework’s performance to this component. At $F_D$9, removing IBQD reduces S4 performance to $E_A$0 and MS3 performance to $E_A$1, compared with full TAViS at $E_A$2 on S4 and $E_A$3 on MS3 (Luo et al., 13 Jun 2025).

4. Text-bridged prompting and alignment supervision

The text-bridged design has two parts: hybrid prompting and alignment supervision. Both use text as an intermediate semantic representation rather than direct audio-image matching (Luo et al., 13 Jun 2025).

For the sparse prompt, TAViS constructs pseudo-text embeddings $E_A$4 by encoding audio-derived tokens through $E_A$5, and constructs audio prompt embeddings $E_A$6 to preserve instance-specific details. These are merged into a single sparse prompt:

$E_A$7

where $E_A$8 denotes concatenation.

The audio-to-pseudo-text loss encourages audio-derived tokens to predict meaningful text features:

$E_A$9

For the dense prompt, the image cls token $E_I$0 is repeated and added to every pixel location in the SAM2 image embedding. The stated intuition is that sparse prompting stabilizes semantics by converting audio to pseudo-text, while dense prompting injects per-pixel context consistent with ImageBind’s shared space.

Alignment supervision then aligns audio-to-text and image-to-text separately. The reported rationale is that direct audio-image alignment is noisy due to intra-class variability and contextual diversity, whereas text provides concise, high-level prototypes. For audio-to-text classification,

$E_I$1

where $E_I$2 is obtained via Hungarian matching.

For image-to-text classification,

$E_I$3

The framework does not use InfoNCE. Instead, it uses cross-entropy over similarity matrices computed by matrix multiplication in the shared projected space. A direct audio-to-image term $E_I$4 was tested and found harmful due to noisy tokens; the final model uses only $E_I$5 and $E_I$6.

Ablations support this formulation. Removing text-bridged alignment supervision reduces S4 to $E_I$7 and MS3 to $E_I$8; removing text-bridged hybrid prompting reduces S4 to $E_I$9 and MS3 to $E_T$0. Within the alignment losses, the final configuration $E_T$1 outperforms alternatives including $E_T$2 (Luo et al., 13 Jun 2025).

5. Optimization, datasets, and empirical results

The segmentation objective combines an object-level loss $E_T$3 over $E_T$4 masks, using Binary Cross-Entropy and IoU losses, with a binary image-level loss $E_T$5 on a combined binary mask. The total loss is

$E_T$6

The training pipeline samples paired frames and audio from each video. For S4 and MS3, the protocol follows AVSBench conventions: S4 has 1 annotated frame per 5, and MS3 has 5 annotated frames per video. Optimization uses Adam with initial learning rate $E_T$7 and cosine decay. Batch sizes are 4 for MS3 and AVSS, and 10 for S4. Epoch counts are 80 for MS3 and S4, and 40 for AVSS. Input sizes are $E_T$8 or $E_T$9, with the paper explicitly noting that higher resolution yields better performance in SAM/SAM2 regimes (Luo et al., 13 Jun 2025).

The evaluation covers AVSBench-object and AVSBench-semantic. AVSBench-object includes S4, a single-source benchmark with 4,932 videos and 1 annotated frame per 5, and MS3, a multi-source benchmark with 424 videos and 5 annotated frames per video. AVSBench-semantic introduces V2 with 6,000 videos and 10 frames per audio clip, and adds class labels to S4 and MS3 to form V1S and V1M. Binary segmentation is measured by mean Jaccard $I \in \mathbb{R}^{T\times H\times W\times C}$0 and mean F-score $I \in \mathbb{R}^{T\times H\times W\times C}$1, while semantic AVS uses class-aware $I \in \mathbb{R}^{T\times H\times W\times C}$2.

The main reported results for the ViT-L backbone are summarized below.

Setting	Dataset	Result
224×224	S4	$I \in \mathbb{R}^{T\times H\times W\times C}$3 84.8, $I \in \mathbb{R}^{T\times H\times W\times C}$4 0.912
224×224	MS3	$I \in \mathbb{R}^{T\times H\times W\times C}$5 68.2, $I \in \mathbb{R}^{T\times H\times W\times C}$6 0.759
224×224	AVSS	$I \in \mathbb{R}^{T\times H\times W\times C}$7 44.2
1024×1024	S4	$I \in \mathbb{R}^{T\times H\times W\times C}$8 87.0, $I \in \mathbb{R}^{T\times H\times W\times C}$9 0.926
1024×1024	MS3	$F_E$0 71.2, $F_E$1 0.796

At $F_E$2, the paper reports S4 $F_E$3 84.8 and $F_E$4 0.912, described as best $F_E$5 and second-best $F_E$6 versus COMBO’s 0.919; MS3 $F_E$7 68.2 and $F_E$8 0.759, reported as state of the art; and AVSS $F_E$9 44.2, also reported as state of the art among reported methods. At $F_D$0, performance rises to S4 $F_D$1 and MS3 $F_D$2.

The paper also gives a fair comparison with SAM-based systems. With SAM-H at $F_D$3, TAViS achieves S4 $F_D$4 and MS3 $F_D$5 at 334G MACs, compared with SAMA-AVS at $F_D$6 and 598G MACs achieving S4 $F_D$7 and MS3 $F_D$8. With SAM2-L at $F_D$9, TAViS achieves $F_M$0 and $F_M$1, surpassing SAMA-AVS at $F_M$2 and $F_M$3 (Luo et al., 13 Jun 2025).

6. Zero-shot behavior, interpretation, and limitations

TAViS is also evaluated in zero-shot and open-vocabulary settings. On AVSBench-OV, the framework reaches $F_M$4 with 54.9M trainable parameters, outperforming OV-AVSS, which reports 22.20 with 183.6M parameters (Luo et al., 13 Jun 2025). In this setting, classification is performed by similarity between text prototypes and both audio and image embeddings, combined by addition with coefficient 1.

The analysis in the source attributes the framework’s gains to three factors. First, the text bridge reduces intra-class noise by anchoring audio and image to compact textual prototypes; t-SNE is reported to show tighter clusters when the text bridge is used. Second, IBQD preserves ImageBind-aligned audio space during source decomposition. Third, the dense visual prompt $F_M$5 injects global context per pixel consistent with the shared space, sharpening mask boundaries.

Several limitations are stated explicitly. Direct audio-image alignment using $F_M$6 harms performance due to noisy tokens, which reinforces the importance of text as a bridge. Performance is strongly dependent on image resolution in SAM and SAM2 regimes; $F_M$7 substantially improves results, but at higher computational cost, and the authors report not running $F_M$8 on AVSS due to resource constraints. Finally, both ImageBind and the SAM2 image encoder are frozen, which stabilizes training but may limit task-specific adaptation.

The paper suggests several plausible extension directions. More robust pseudo-text generation or denoising strategies could mitigate token noise and perhaps enable safe audio-image alignment. Efficient high-resolution training or inference could retain the benefits of $F_M$9 at lower compute. Broader zero-shot capability might be obtained through richer text prompting or external lexical knowledge. These are presented as implications of the reported findings rather than established results.