---
title: 'AudioFuse: Dual-Branch Audio Fusion'
url: https://www.emergentmind.com/topics/audiofuse
type: topic
---

# AudioFuse: Dual-Branch Audio Fusion

AudioFuse most directly denotes a dual-branch architecture for biomedical audio classification that jointly learns from a **2D log-Mel spectrogram** and the **raw 1D waveform** in order to classify phonocardiograms (PCGs) [2509.23454]. More broadly, the term is also useful as a shorthand for a recurrent design pattern in audio machine learning: combining complementary representations or modalities—such as intermediate and final acoustic features, audio and video, or speech spectra and listener profiles—through explicit fusion mechanisms rather than relying on a single representation alone [2210.05037]. In that broader sense, AudioFuse refers less to one universal algorithm than to a class of architectures organized around representational complementarity, controlled fusion, and task-specific decoding.

## 1. Definition and scope

In its most specific usage, **AudioFuse** is the model introduced in "AudioFuse: Unified Spectral-Temporal Learning via a Hybrid ViT-1D CNN Architecture for Robust Phonocardiogram Classification" [2509.23454]. That model is a lightweight dual-branch deep network for **phonocardiogram (PCG) classification** that processes a **2D log-Mel spectrogram** with a custom **Vision Transformer (ViT)** and the **raw 1D waveform** with a shallow **1D CNN**, then combines the resulting feature vectors with **late fusion**. The central claim is that biomedical audio such as PCG is inherently both **spectral** and **temporal/rhythmic**, so a model that only sees a spectrogram or only sees a waveform is intrinsically incomplete [2509.23454].

The same underlying logic appears in other audio research, although often under different names. In automated audio captioning, the **Low- and High-Dimensional Feature Fusion (LHDFF)** model argues that intermediate CNN activations may retain complementary acoustic detail that is useful for caption generation, and therefore fuses an intermediate “low-dimensional” representation with the final “high-dimensional” representation of a PANNs/CNN10 encoder [2210.05037]. In audio-visual event recognition, **MAFnet** combines intermediate cross-modal conditioning through a FiLM-based lateral connection with high-level modality-temporal attention, thereby performing fusion at more than one representational level [2106.06736]. In personalized speech enhancement for hearing aids, **AFN-HearNet** performs **middle fusion** of encoded speech spectra and an audiogram-derived latent representation through affine modulation inside frequency-temporal Conformer blocks [2509.07341].

This suggests that “AudioFuse” can be understood in two senses. First, it is the proper name of a specific PCG classifier [2509.23454]. Second, it is a plausible umbrella label for architectures whose core inductive bias is that useful audio information is distributed across heterogeneous but complementary representations, and that explicit fusion is therefore preferable to single-stream processing [2210.05037].

## 2. Representational complementarity as the core principle

The explicit motivation of AudioFuse for PCG classification is representational complementarity. A raw discrete audio signal preserves precise event timing, waveform morphology, phase information, and local rhythmic patterns, whereas a spectrogram provides explicit time-frequency organization and strong visibility of tonal and harmonic patterns [2509.23454]. The paper frames waveform and spectrogram as **complementary rather than competing** representations, because each exposes information the other makes less accessible.

Within this argument, the spectrogram branch is useful because it makes frequency structure directly visible, but it also imposes the usual STFT trade-off and, as stated in the paper, “discards all phase information” and “blurs temporal details” [2509.23454]. The waveform branch preserves exact timing and phase, but requires the model to infer frequency structure indirectly. AudioFuse addresses this mismatch by jointly modeling both representations rather than privileging one.

A closely related argument appears in AAC. The LHDFF model states that most strong AAC systems use a pretrained audio encoder such as PANNs/CNN10 and then feed only its final, high-dimensional representation into a language decoder, but this may discard useful intermediate information [2210.05037]. The proposed solution is to retain and fuse the output of the **third convolutional block** with the final-layer output, on the premise that intermediate CNN activations may preserve complementary acoustic detail that final abstract embeddings suppress [2210.05037].

The same principle extends beyond single-modality multi-representation fusion. In **MAFnet**, the relevance of each modality can change over time within a video, so the model should dynamically answer both **which time window matters most** and **which modality matters most at that time** [2106.06736]. In **AFN-HearNet**, the two inputs are not two views of the same signal but a dynamic speech spectrum and a static sparse audiogram; fusion is motivated by the claim that direct concatenation is inadequate because the two inputs are misaligned in both **resolution** and **structure** [2509.07341]. Across these cases, fusion is justified by complementarity, mismatch, or both.

## 3. Architecture of AudioFuse for phonocardiogram classification

AudioFuse for PCG classification consists of two synchronized input pathways derived from the same recording [2509.23454]. One pathway constructs a **2D log-Mel spectrogram** of size **\(224 \times 224\)** and processes it with a custom **wide-and-shallow Vision Transformer**. The other takes the **raw 1D waveform**, padded or truncated to a fixed length of **5 seconds**, corresponding to **110,250 samples**, and processes it with a shallow **1D CNN** [2509.23454].

The spectrogram branch divides the **\(224 \times 224\)** input into **196 patch embeddings**, corresponding to a **\(14 \times 14\)** grid and therefore a **\(16 \times 16\)** patch size. Patch extraction is implemented via a **Conv2D** layer with kernel size **\(16 \times 16\)**, and the embedding dimension is **192**. The branch adds **learnable positional embeddings**, then applies **6 Transformer encoder blocks** with **8 attention heads** and an MLP, and finally globally averages the output token sequence to obtain a spectral feature vector \(\mathbf{f}_{spec} \in \mathbb{R}^{192}\) [2509.23454].

The waveform branch consists of **three sequential Conv1D blocks**, each using kernel size **16** and stride **4**, followed by **Batch Normalization** and **MaxPooling**. The number of filters increases from **64** to **128** to **256** across the three blocks. After the final convolution block, the feature sequence is globally averaged and passed through a final **Dense layer with 64 units**, yielding \(\mathbf{f}_{wave} \in \mathbb{R}^{64}\) [2509.23454].

Fusion is performed by simple late concatenation:
\[
\mathbf{f}_{fused} = [\mathbf{f}_{spec}; \mathbf{f}_{wave}]
\]
with \(\mathbf{f}_{fused} \in \mathbb{R}^{256}\) [2509.23454]. The fused vector is passed through a small MLP head consisting of **Dense(192)**, **Dropout(0.5)**, and a final **sigmoid** output layer [2509.23454]. The paper also studies **Cross-Attention**, **Gated FiLM**, and **Spectrogram-Scalogram fusion**, but the core AudioFuse design emphasized in the architecture section is concatenation-based late fusion, which also performs best in the reported comparison [2509.23454].

## 4. Mathematical formulation and fusion mechanisms across related systems

For AudioFuse, the spectrogram branch is grounded in the Short-Time Fourier Transform:
\[
X(t, \omega) = \sum_{n=-\infty}^{\infty} x[n] w[n-t] e^{-j\omega n}
\]
with the power spectrogram defined as
\[
P(t,\omega) = |X(t,\omega)|^2
\]
before mapping to the Mel scale [2509.23454]. At the representation level, fusion is expressed directly as concatenation of the branch embeddings:
\[
\mathbf{f}_{fused} = [\mathbf{f}_{spec} ; \mathbf{f}_{wave}]
\]
which preserves branch specialization up to a late decision stage [2509.23454].

The broader AudioFuse pattern includes several distinct fusion operators. In LHDFF for AAC, the encoder output from the third convolutional block is treated as the “low-dimensional” feature \(x_3 \in \mathbb{R}^{T' \times 256}\), while the final-layer output is the “high-dimensional” feature \(x_{final} \in \mathbb{R}^{T \times 1024}\) [2210.05037]. Each is projected to 128 dimensions with ReLU:
\[
x_{high} = Relu(f_{128}(x_{final})), \quad x_{high}\in \mathbb{R}^{T\times 128};
\]
\[
x_{low} = Relu(f_{128}(x_{3})), \quad x_{low}\in \mathbb{R}^{T^{'}\times 128};
\]
\[
x_{fusion} = x_{high} \oplus x_{low}, \quad x_{fusion}\in \mathbb{R}^{T\times 128}.
\]
Because \(T' \neq T\), the authors state that **“the method of patch 0 is used to make the dimension of \(T'\) and \(T\) the same”**, i.e., zero-padding along the temporal axis before fusion [2210.05037]. The text does not define \(\oplus\) precisely, but the reported output shape implies a same-shape merge rather than concatenation.

On the decoder side, LHDFF uses two Transformer decoders, one conditioned on \(x_{fusion}\) and one on \(x_{high}\), and fuses the resulting token distributions:
\[
P_{fusion} = P_{TD_{1} \oplus P_{TD_{2}, \quad P_{fusion}\in \mathbb{R}^{T\times m}.
\]
Again, \(\oplus\) is not explicitly defined, but the figure caption states that “the final fused probability result is obtained by fusing the log softmax probability distribution of each transformer decoder” [2210.05037]. This locates fusion at the token-distribution level rather than the feature level.

In **MAFnet**, fusion is explicitly multi-level. At the intermediate level, visual features modulate audio feature maps through FiLM:
\[
FiLM(F^2_{t,c}|\gamma_{t,c}, \beta_{t,c}) = \gamma_{t,c} F^2_{t,c} +  \beta_{t,c}
\]
where \(\gamma_{t,c}\) and \(\beta_{t,c}\) are generated from the visual representation [2106.06736]. At the high level, the model computes joint modality-temporal attention over modality-time pairs:
\[
\lambda_t^k = \frac{\exp(z_t^k)}{\sum_{j=1}^{T} \sum_{l=1}^{K} \exp(z_j^l)}
\]
and forms the global fused representation through weighted temporal pooling within each modality followed by concatenation across modalities [2106.06736].

In **AFN-HearNet**, fusion takes the form of affine modulation in latent space rather than concatenation or cross-attention. The encoded speech feature is modulated by audiogram-derived parameters:
\[
Z^\prime_f = \left[\mathcal{T}\left(Z_f\odot\left(I+\gamma_1\right)+\beta_1\right)\right]\odot\alpha_1+Z_f,
\]
followed by a second modulation and MLP step:
\[
\hat{Z}_f = \left[MLP\left(Z^\prime_f\odot\left(I+\gamma_2\right)+\beta_2\right)\right]\odot\alpha_2+Z^\prime_f.
\]
The paper characterizes this as **middle fusion**, **feature-wise conditional modulation**, and **FiLM-like affine modulation plus gating** [2509.07341].

## 5. Empirical performance and comparative results

On the **PhysioNet 2016** dataset, the PCG AudioFuse model reports **ROC-AUC \(0.8608 \pm 0.0127\)** when trained **from scratch**, compared with **0.8066** for the spectrogram-only baseline and **0.8223** for the waveform-only baseline [2509.23454]. In the same table, **AudioFuse (Concatenation)** reports **Accuracy \(0.7741 \pm 0.0094\)**, **F1 \(0.7664 \pm 0.0005\)**, **ROC-AUC \(0.8608 \pm 0.0127\)**, and **MCC \(0.5508 \pm 0.0225\)** with **2.56M** parameters [2509.23454]. The paper also reports **AudioFuse (Cross-Attention)** at **ROC-AUC \(0.8226 \pm 0.0161\)** and **AudioFuse (Gated FiLM)** at **ROC-AUC \(0.8518 \pm 0.0141\)**, establishing that the simplest fusion operator performs best in this setting [2509.23454].

For out-of-domain evaluation on **PASCAL Set B**, AudioFuse reports **Accuracy 0.7386**, **F1 0.6667**, **ROC-AUC 0.7181**, and **MCC 0.4519**, compared with **ROC-AUC 0.4873** for the spectrogram-only ViT baseline and **0.6782** for the waveform-only 1D CNN baseline [2509.23454]. The paper interprets this as evidence that fusion improves resilience to domain shift and that temporal features are more robust than spectrogram-only features under this particular shift [2509.23454].

Comparable but task-specific gains are reported in AAC. On **Clotho**, the baseline without reinforcement learning reports **SPIDEr 0.256**, while LHDFF reaches **0.265**; on **AudioCaps**, the baseline SPIDEr is **0.404**, **Pre-BERT** gives **0.410**, **AT-CNN10** gives **0.414**, and LHDFF reaches **0.426** [2210.05037]. The ablations are especially important: **“LHDFF (only fusi-fea)”** reaches **SPIDEr 0.261** on Clotho and **0.419** on AudioCaps, supporting the claim that the intermediate low-dimensional feature contributes complementary information [2210.05037].

In audio-visual event recognition, **MAFnet** reports **90.86** accuracy on **AVE**, **86.72** on **UCF51**, and **83.94** on **Kinetics-Sounds** [2106.06736]. Its ablation on AVE shows **concatenation: 89.34**, outperforming **addition: 84.77**, **MCB: 88.83**, and **DMR: 87.56**, while the best conditioning direction is **visual \(\rightarrow\) audio**, yielding **90.86** [2106.06736]. In personalized speech enhancement, **AFN-HearNet** reports on the **DNS-Challenge test set**: **1.28M parameters**, **3.39 G FLOPs/s**, **HASQI 0.81**, **WB-PESQ 2.75**, **NB-PESQ 3.29**, **SDR 13.86**, **SI-SNR 13.62**, and **STOI 95.55**; the paper states that AFN-HearNet significantly outperforms state-of-the-art in-context fusion joint models regarding key metrics such as HASQI and PESQ [2509.07341].

These results do not establish a single universal superiority of one fusion operator. Instead, they show that the empirical value of AudioFuse-style design depends on the task, the representational mismatch being addressed, and the location at which fusion occurs.

## 6. Design patterns, limitations, and interpretation

Several recurring design patterns emerge across AudioFuse-related systems. One is **late fusion by concatenation**, which is the main mechanism in the PCG AudioFuse model and also the strongest fusion operator in MAFnet’s AVE ablation [2509.23454]. Another is **intermediate or middle fusion**, as in LHDFF’s combination of intermediate and final CNN features and AFN-HearNet’s affine modulation of encoded speech features by an encoded audiogram [2210.05037]. A third is **distribution-level fusion**, as in LHDFF’s merging of log-softmax outputs from two Transformer decoders [2210.05037].

A second recurring pattern is that explicit fusion is often paired with architectural restraint. AudioFuse for PCG classification uses a deliberately **wide-and-shallow ViT**, a shallow **1D CNN**, late fusion, **dropout 0.5**, and **early stopping**, with a total parameter count of **2.56M** for the concatenation model [2509.23454]. The paper argues that these choices mitigate the overfitting risk common in fusion models. Likewise, AFN-HearNet discusses efficiency and hearing-aid suitability explicitly, reporting a **theoretical latency of 16 ms** under causal constraints and noting the possibility of replacing the fusion module with lighter sequence modules [2509.07341].

The literature also makes clear that fusion is not identical to general multimodality. LHDFF is explicitly **intra-audio, multi-level feature fusion**, not cross-modal audio-text fusion inside the encoder [2210.05037]. AFN-HearNet is an audio fusion model in a specialized sense: it fuses a dynamic speech spectrum with a static hearing-loss profile, not audio with video or text [2509.07341]. MAFnet, by contrast, is genuinely audiovisual, but still structured around the idea that fusion should happen at more than one level [2106.06736].

The limitations are equally consistent. The PCG AudioFuse paper does not report many implementation specifics for spectrogram generation or some branch internals, which limits exact reproducibility from the text alone [2509.23454]. LHDFF leaves several details ambiguous, including the exact meaning of \(\oplus\), the number of Transformer layers, positional encoding details, and whether PANNs is pretrained and frozen or finetuned [2210.05037]. AFN-HearNet is more explicit about the fusion mechanism, but remains specialized to hearing-aid speech enhancement rather than a generic audio fusion setting [2509.07341].

This suggests that “AudioFuse” is best interpreted not as a settled canonical architecture, but as a family of explicit fusion strategies whose success depends on careful matching between the representational split and the downstream task.

## 7. Broader significance and related directions

The significance of AudioFuse-style work lies in the claim that architectural bias can substitute, at least partly, for scale or pretraining when the task structure is well understood. The PCG AudioFuse paper emphasizes that the model is trained **from scratch** and does **not** rely on large-scale pretraining, yet remains competitive with substantially larger spectrogram models while being much smaller than **DenseNet169** and **InceptionV3** [2509.23454]. The practical message is that, for rhythmic biomedical audio, the joint use of spectral and temporal views can provide a strong inductive bias.

A related but more data-centric direction appears in **FusionAudio-1.2M**, which uses a multimodal contextual fusion pipeline to generate **1.2M detailed captions** and **6M QA pairs** from AudioSet-derived clips [2506.01111]. Fusion there is not a learned dual-branch classifier but a two-stage automated pipeline that extracts expert cues from multiple modalities—speech, music, general sounds, and visual context—and synthesizes them with an LLM into detailed audio-only captions [2506.01111]. The paper reports that more than **50% of samples** integrate **two or more modalities**, and that removing video, speech, music, or filtering reduces downstream performance [2506.01111]. This indicates that fusion can also function at the level of data construction and supervision rather than only within the final task model.

Another extension is modular infrastructure for hybrid audio systems. **NablAFx** is presented as an open-source PyTorch framework for differentiable audio effect modeling, supporting black-box and gray-box models, conditioning methods such as **FiLM**, **TFiLM**, **TTFiLM**, **TVCond**, and **TVFiLM**, and differentiable DSP blocks [2502.11668]. Although it is not an AudioFuse paper, it is highly relevant as a substrate for systems that would combine learned waveform processors, controller-mediated conditioning, and explicit DSP modules [2502.11668]. The paper explicitly notes that hybrid models combining black-box and gray-box processors remain future work, which is a plausible direction for more general AudioFuse-like systems [2502.11668].

A further frontier is the unification of heterogeneous front-end tasks in conversational audio systems. **UAF** reformulates **VAD**, **turn-taking detection**, **speaker recognition**, **ASR**, and **QA** as a single auto-regressive sequence prediction problem over semantic and system-control tokens [2604.19221]. Although not labeled AudioFuse, it is directly analogous in that it replaces multiple front-end modules with one shared representation and specialized lightweight heads [2604.19221]. This suggests that the logic of AudioFuse may extend from representational fusion within a signal to functional fusion across tasks.

Taken together, these works indicate that AudioFuse is both a specific PCG architecture and a more general research orientation. In the narrow sense, it denotes a **hybrid ViT–1D CNN** classifier for heart sounds [2509.23454]. In the broader sense, it names a design philosophy: preserve complementary information, align heterogeneous representations carefully, and fuse them at a level appropriate to the task rather than assuming that one representation, one modality, or one decoder branch is sufficient [2210.05037].

Source: https://www.emergentmind.com/topics/audiofuse