Audio-Aware Query-Enhanced Transformers (AuTR)

• The paper introduces AuTR, a multimodal transformer that seeds its decoder with audio-derived queries to focus on sounding objects in video frames.
• It employs modality-specific encoders and a unified transformer backbone, achieving improved Jaccard and F-scores over previous convolutional fusion methods.
• The architecture incorporates dynamic convolution in the segmentation head to enhance cross-modal binding and support robust performance in multi-sound and open-set scenarios.

Audio-Aware Query-Enhanced Transformers (AuTR) are a class of multimodal neural architectures specifically designed for the task of audio-visual segmentation (AVS). AVS aims to segment the regions in a video frame that correspond to sounding objects, leveraging both audio and visual cues. AuTR addresses limitations of prior convolutional and early-fusion approaches—namely their restricted receptive fields and inadequate modality fusion—by introducing a unified multimodal transformer framework seeded with audio-derived queries. This architecture enables explicit disambiguation of sounding versus silent (but salient) objects, more robust cross-modal binding, and improved generalization in multi-sound and open-set scenarios (Liu et al., 2023).

1. Architectural Overview

The AuTR framework is organized into three consecutive processing stages: (1) modality-specific encoders for visual and audio streams, (2) a multimodal transformer backbone with both encoder and a query-enhanced decoder, and (3) a lightweight segmentation head based on dynamic convolution.

• Visual Encoder: A pre-trained ResNet-50 or Pyramid Vision Transformer (PVT-v2) consumes RGB frames $v \in \mathbb{R}^{3 \times H_0 \times W_0}$, yielding three visual feature maps at spatial resolutions $$f^t_{v,\ell} \in \mathbb{R}^{C_v \times H_\ell \times W_\ell}$$ for $\ell=1,2,3$.
• Audio Encoder: A VGGish backbone (pre-trained on AudioSet) converts $T \times H_a \times W_a$ log-mel spectrograms into $C_a$-dimensional embeddings $f^t_a$.
• Audio–Visual Early Fusion: Both visual and audio embeddings are linearly mapped to a common dimension $C_{av}$ for each scale $\ell$. Multi-head attention fuses these as follows:

$$Q = W_Q f^t_{v,\ell},\quad K = W_K f^t_a,\quad V = W_V f^t_a$$

$$\mathrm{Attention}_\ell(f^t_{v,\ell}, f^t_a) = \mathrm{softmax}\left(\frac{QK^T}{\sqrt{d}}\right)V$$

This produces three audio–visual fused feature maps $$f^t_\ell \in \mathbb{R}^{C_{av} \times H_\ell \times W_\ell}$$.

2. Multimodal Transformer Backbone

2.1 Multimodal Encoder

The fused feature maps are spatially flattened, scale-stacked, temporally stacked, and enhanced with fixed 2D positional encodings. They are then processed by a 6-layer deformable transformer encoder, enabling self-attention and cross-scale attention to yield deeply aggregated representations $F'_{av}$.

2.2 Audio-Aware Query-Enhanced Decoder

AuTR displaces traditional DETR-style object queries with $N_q$ learnable vectors $Q^0_i \in \mathbb{R}^D$ ($i=1 \dots N_q$), each initialized to $f_a + \text{pos}_i$ (audio embedding plus positional offset). Decoder operation comprises, per layer $\ell$: 1. Self-attention among current queries: $$Q^{\ell'} = \mathrm{Attn}(Q^{\ell-1}, Q^{\ell-1}, Q^{\ell-1})$$. 2. Cross-attention using the encoded audio-visual tokens: $$Q^{\ell''} = \mathrm{Attn}(Q^{\ell'}, K=F'_{av}, V=F'_{av})$$. 3. Feed-forward update with residual: $Q^{\ell} = \mathrm{FFN}(Q^{\ell''}) + Q^{\ell''}$.

This explicit seeding of the decoder with audio semantics ensures the attention mechanism focuses on modality-coherent (i.e., sounding) regions.

3. Deep Aggregation and Mask Head

A parallel lightweight decoder, following the Feature Pyramid Network (FPN) paradigm, aggregates the original visual features ($f^t_{v,1..3}$), audio tokens ($f^t_a$), and transformer encoder output ($F'_{av}$) via cross-attention and progressive top-down upsampling, generating a multi-modal feature map $F_m \in \mathbb{R}^{C_m \times H_m \times W_m}$.

The $N_q$ decoder output vectors are passed through a small MLP, generating per-query convolution kernels $k_i \in \mathbb{R}^{D_s}$, consisting of $1 \times 1$ and $3 \times 3$ kernel weights. This results in:

$\hat{S}_i = F_m \ast k_i$

where each $\hat{S}_i$ is a low-resolution mask for query $i$.

4. Training Objectives and Optimization

For each training frame, only a single ground-truth mask $y$ (the sounding object) is provided. A bipartite matching cost is computed:

$$i^* = \arg\min_i\left[ \lambda_d C_\text{dice}(\hat{Y}_i, y) + \lambda_f C_\text{focal}(\hat{Y}_i, y) + \lambda_s C_\text{sound}(s_i, s_{gt}) \right]$$

with $C_\text{dice}$ (soft Dice loss), $C_\text{focal}$ (binary focal loss), and $C_\text{sound}$ (binary cross-entropy for sounding-score head $s_i$).

The total loss is applied only to the matched query:

$$\mathcal{L} = \lambda_d \mathrm{Dice}(\hat{Y}_{i^*}, y) + \lambda_f \mathrm{Focal}(\hat{Y}_{i^*}, y) + \lambda_s \mathrm{BCE}(s_{i^*}, 1)$$

Typical loss weights: $\lambda_d=1$, $\lambda_f=1$, $\lambda_s=0.1$.

Key hyperparameters:

• $N_q = 100$ (audio-aware queries)
• Transformer: 6 encoder and 6 decoder layers, 8 attention heads, $D=256$ hidden dimension
• Feature dims: $C_{av}=256$, $C_m=128$, $D_s \approx 1000$
• AdamW optimizer: learning rate $1 \times 10^{-4}$, weight decay $5 \times 10^{-4}$; 50 epochs on a single NVIDIA RTX 3090 (batch size 8, modality backbones frozen)

5. Empirical Results

Quantitative performance is assessed on AVSBench S4 (single sound) and MS3 (multi-sound) benchmarks as well as open-set evaluation. On S4 with ResNet-50:

• TPAVI achieves $(J/F) = 72.8 / .848$; AuTR reaches $75.0 / .852$ $(+2.2/+0.4)$.
• With PVT-v2: TPAVI at $78.7 / .879$, AuTR at $80.4 / .891$. For MS3 (multi-sound) without fine-tuning (ResNet-50): TPAVI at $47.9 / .578$, AuTR at $49.4 / .612$. After S4$\rightarrow$MS3 fine-tuning, TPAVI drops from $47.9$ to $44.3$, while AuTR improves from $49.4$ to $56.0$ $(+6.6)$.

Open-set results (PVT-v2):

• Seen: AuTR $77.56/.865$, TPAVI $75.62/.862$
• Unseen: AuTR $66.22/.777$, TPAVI $55.86/.719$

Ablation results:

• Removing audio-aware queries: Jaccard drops from $80.4$ to $79.6$.
• Removing dynamic-conv mask head: $79.5$.

Qualitative assessment reveals AuTR's ability to isolate truly sounding objects even in cluttered, multi-object, or unseen-category scenes, whereas TPAVI often selects silent but salient distractors.

6. Distinctive Features and Comparative Analysis

The explicit integration of audio signals as the initial state for transformer queries results in focused and interpretable attention; visual–spatial areas emitting sound are emphasized, while equally salient but soundless objects are suppressed. The combination of dynamic convolution in the segmentation head and deep transformer-based cross-modal fusion distinguishes AuTR from prior convolutional fusion methods.

Earlier fusion-based AVS approaches are limited by small receptive fields and superficial fusion. In contrast, AuTR's transformer stack, multi-scale aggregation, and learned audio query initialization allow for more comprehensive cross-modal reasoning and improved domain adaptation, particularly in multi-sound and open-set regimes.

7. Pseudocode and Workflow Illustration

A schematic summary of AuTR’s per-frame inference loop is as follows:

f_v1, f_v2, f_v3 = VisualBackbone(v_t) f_a = AudioBackbone(a_t) for ℓ in {1,2,3}: f_avℓ = MultiHead(Proj_v(f_vℓ), Proj_a(f_a), Proj_a(f_a)) F′_av = TransformerEncoder(flatten(f_av1, f_av2, f_av3)) Q^0 = [f_a + pos_i for i in 1…N_q] for ℓ in 1…L_decoder: Q′ = SelfAttn(Q^{ℓ−1}) Q″ = CrossAttn(Q′, keys=F′_av, vals=F′_av) Q^ℓ = FFN(Q″) F_m = PixelDecoder(f_v1, f_v2, f_v3, f_a, F′_av) {k_i} = MLP(Q^L) {Ŝ_i} = DynamicConv(F_m, {k_i}) match i* = Hungarian({Ŝ_i}, y_t) compute L_t w.r.t. Ŝ_{i*} and sound‐score s_{i*}

A plausible implication is that further exploration of modality-aware query initialization and adaptive fusion mechanisms could extend AuTR’s robustness to additional domains, such as cross-modal retrieval or active sound-source localization (Liu et al., 2023).