Two-Stream Attention in Audio Tagging

• The paper introduces two-stream attention as neural architectures that fuse complementary feature streams via attention mechanisms for improved audio event classification.
• It employs global-local, multi-feature, and multi-modal strategies to combine distinct temporal and acoustic cues for robust audio tagging.
• Empirical evaluations demonstrate that dynamic fusion and joint training yield consistent performance gains in weakly supervised audio tagging tasks.

Two-stream attention in audio tagging refers to neural frameworks that process and fuse multiple sources or levels of information—most commonly, distinct feature subsets (e.g., Mel and log-Mel spectrograms), different temporal resolutions (“global” vs. “local” context), or even different modalities (e.g., audio and visual)—through attention-based interaction mechanisms. These architectures are designed to address the limitations of single-stream approaches, especially in weakly supervised scenarios, by leveraging complementary properties across streams for improved classification and localization of sound events.

1. Two-Stream Architectures: Conceptual Overview

In the domain of audio tagging, two-stream models are network architectures that feed two separate but parallel pipelines (“streams”) with distinct but related input features or temporal segments. Attention-based modules are then deployed to facilitate interaction between streams, either at early, intermediate, or late network stages. This enables selective feature alignment, information transfer, and aggregation, allowing models to exploit both broad and fine-grained cues as well as multi-modal correlations. Distinctions exist regarding the nature of the two streams:

• Global-local attention architectures treat one stream as operating on the entirety of the clip and another on selected sub-clips (Wang et al., 2021).
• Multi-feature or multi-modal attention architectures process different acoustic features, or combine spectro-temporal and visual inputs (Bhatt et al., 2018, Boes et al., 2022).

2. Global-Local Attention Frameworks

The global-local attention framework (GL-AT) exemplifies the two-stream paradigm for weakly labelled audio tagging, as described in “A Global-local Attention Framework for Weakly Labelled Audio Tagging” (Wang et al., 2021).

• Input: A 10-second audio clip $A \in \mathbb{R}^{T_0 \times S}$.
• Global stream: Extracts frame-wise features $M \in \mathbb{R}^{T \times C}$, pools them to obtain a clip-level feature $M' \in \mathbb{R}^{1 \times C}$, and makes a global prediction $\hat{y}_g \in [0,1]^L$ via sigmoid activation.
• Class-wise selection: The classifier is applied frame-wise to $M$, yielding $S_g \in [0,1]^{T \times L}$ class activations. For the top $N$ classes, time indices of peak activation are used to extract $N$ local sub-clips of length $\tau$.
• Local stream: Processes each sub-clip $A_k$ through the same feature extractor and classifier, predicts $M \in \mathbb{R}^{T \times C}$0, and pools over $M \in \mathbb{R}^{T \times C}$1 to yield a local prediction $M \in \mathbb{R}^{T \times C}$2.
• Inference fusion: Final predictions combine $M \in \mathbb{R}^{T \times C}$3 and $M \in \mathbb{R}^{T \times C}$4, typically via averaging or custom pooling.
• Training: Both streams are trained jointly under weak (clip-level) supervision with the sum of two binary cross-entropy losses. Backpropagation affects both streams, guided by the differentiable sub-clip selection process.

This design exploits both global context (for robust initial localization) and local temporal precision (for fine-grained decision-making), as evidenced by ablation studies.

3. Multi-Feature and Multi-Modal Two-Stream Attention

Beyond global-local splits, two-stream architectures have been developed to combine multiple feature types or modalities. Notable instantiations are found in the attentive multi-channel fusion of Mel and log-Mel spectrograms (Bhatt et al., 2018) and audio-visual Transformer models (Boes et al., 2022).

Multi-Feature (Mel/Log-Mel)

• Inputs: Parallel streams receive Mel-spectrogram and log-Mel-spectrogram features, both with 40 Mel filters.
• Stream pipelines: Each follows a shared-weights two-layer CNN stack ($M \in \mathbb{R}^{T \times C}$5 then $M \in \mathbb{R}^{T \times C}$6 filters), global-average-pooled to $M \in \mathbb{R}^{T \times C}$7-dim embeddings.
• Early attention-based fusion: At intermediate convolutional layers, a similarity-based attention matrix $M \in \mathbb{R}^{T \times C}$8 is computed between per-frame activations of each stream:

$M \in \mathbb{R}^{T \times C}$9

Attentive features $M' \in \mathbb{R}^{1 \times C}$0 modulate each stream's intermediate representation based on what the other stream “attends.”

• Late fusion: The per-segment embeddings $M' \in \mathbb{R}^{1 \times C}$1, $M' \in \mathbb{R}^{1 \times C}$2 (from each stream) are combined via a bi-linear interaction:

$M' \in \mathbb{R}^{1 \times C}$3

These, together with $M' \in \mathbb{R}^{1 \times C}$4, feed into an MLP fusion block for final multi-label prediction.

Multi-Modal (Audio-Visual)

• Encoders: Separate encoder stacks for audio (log-mel spectrograms via a 7-block CNN) and visual (pretrained OpenL3 or temporally coherent embedding projections), each producing $M' \in \mathbb{R}^{1 \times C}$5 feature maps.
• Decoder: At each Transformer or Conformer decoder layer, queries attend over the outputs of each encoder via two separate cross-attention blocks:

$M' \in \mathbb{R}^{1 \times C}$6

Stream fusion is performed via convex combination:

$M' \in \mathbb{R}^{1 \times C}$7

$M' \in \mathbb{R}^{1 \times C}$8 weights may be statically optimized, randomly sampled during training to induce robustness, or dynamically collapsed to audio-only if visual is missing.

4. Mathematical Formulation and Attention Mechanisms

The mathematical instantiations of two-stream attention differ across architectures:

• Global-local selection (Wang et al., 2021): Uses max+avg pooling for temporal aggregation; class-wise frame activations for dynamic routing; binary cross-entropy for joint optimization:

$M' \in \mathbb{R}^{1 \times C}$9

Here, MIL-style training leverages only weak (clip-level) labels.

• Multi-feature attention (Bhatt et al., 2018): Frame-to-frame similarity attention aligns information across streams, affecting CNN-channel weights, followed by bilinear interaction at the segment level.
• Multi-modal cross-attention (Boes et al., 2022): Multi-head attention operates both for self-attention within each stream and cross-attention in the decoder, using standard Transformer primitives but fusing the stream outputs at each layer by weighted summation for robust multi-modal integration.

5. Empirical Performance and Ablation Insights

Global-Local (GL-AT) (Wang et al., 2021)

• Yields consistent 0.02–0.04 mAP improvement across CNN10, ResNet38, TALNet, and AT-SCA backbones on AudioSet, under standard PANNs splits.
• Joint global-local training improves global features by leveraging local-stream guidance.
• The local stream alone outperforms the global-only baseline, demonstrating the effectiveness of focusing on highly activated sub-clips.
• Ablation: Increasing $\hat{y}_g \in [0,1]^L$0 (number of local clips) improves mAP up to $\hat{y}_g \in [0,1]^L$1 (saturates thereafter). Optimal sub-clip duration $\hat{y}_g \in [0,1]^L$2 is 3–4 s, matching empirical sound event durations.

Method	Baseline mAP	+GL-AT mAP	mAUC (→)	d′ (→)
TALNet	0.368	0.401	0.967→0.970	2.600→2.659
CNN10	0.382	0.408	0.969→0.974	2.664→2.742
ResNet38	0.429	0.438	0.974→0.975	2.713→2.774
AT-SCA	0.392	0.413	0.969→0.971	2.658→2.677

Multi-Feature Attention (Bhatt et al., 2018)

• On the CHiME-Home dataset, the attentive bi-channel model yields an equal error rate (EER) of $\hat{y}_g \in [0,1]^L$3, compared to $\hat{y}_g \in [0,1]^L$4 for log-Mel+CRNN and $\hat{y}_g \in [0,1]^L$5 for the vanilla (no attention, no interaction) fusion baseline.
• Early (attention) and late (bi-linear) fusion both contribute to improved error rates; jointly they provide the greatest benefit.

Multi-Modal Attention (Boes et al., 2022)

• Transformer-based audio-only: CBF1 = $\hat{y}_g \in [0,1]^L$6
• Transformer+OpenL3 (bi-encoder): CBF1 = $\hat{y}_g \in [0,1]^L$7
• Gain is robust for segment-based F1 and confirmed for Conformer architectures.

When supplementary data (visual) becomes unavailable, the model dynamically reverts to audio-only fusion with no performance penalty. Adding non-temporal visual features (e.g., VGG16) provides no benefit, and gains diminish when less than 10% of clips include visual input.

6. Training Strategies and Regularization

• GL-AT: Trains both streams with sum of BCE losses, using Adam optimizer. All candidate CNN or CRNN backbones are permitted. Pooling combines max and average operations.
• Attentive Fusion (Bhatt et al., 2018): Adam with L2 regularization, dropout applied to the fusion block; shared CNN weights across streams.
• Multi-Encoder: Rectified Adam, mixup data augmentation. Teacher-student consistency regularization is used in addition to the supervision from weakly and strongly labeled samples. Visual dropout is simulated during training.

7. Significance and Impact

Two-stream attention frameworks provide systematic mechanisms for exploiting heterogeneity in audio, temporal, and associated multi-modal data. GL-AT demonstrates that combining global context with short, high-activation local sub-windows—via class-wise attention and selection—confers consistent lift in weakly supervised tagging, with minimal architectural overhead (Wang et al., 2021). Multi-feature and multi-modal two-stream approaches extend this principle, showing that diverse spectro-temporal and visual cues can be integrated via attention mechanisms for further improvements (Bhatt et al., 2018, Boes et al., 2022). Dynamic fusion mechanisms confer robustness to missing modalities and allow two-stream attention frameworks to be deployed in diverse, real-world multimedia tagging scenarios.

A plausible implication is that future research will increasingly fuse heterogeneous sources—acoustic, visual, and contextual—using modular attention-based two-stream architectures for scalable, robust, and fine-grained classification in noisy, weakly-supervised, or partially observed conditions.