AudioFuse: Dual-Branch Audio Fusion

Updated 13 July 2026

AudioFuse is a dual-branch architecture that fuses 2D log-Mel spectrograms and 1D waveforms to capture complementary spectral and temporal features for PCG classification.
It exemplifies a design pattern in audio machine learning that leverages heterogeneous representations through explicit, controlled fusion techniques.
The system uses concatenation-based late fusion to enhance robustness and performance, as evidenced by improved ROC-AUC and accuracy over single-branch models.

AudioFuse most directly denotes a dual-branch architecture for biomedical audio classification that jointly learns from a 2D log-Mel spectrogram and the raw 1D waveform in order to classify phonocardiograms (PCGs) (Siddiqui et al., 27 Sep 2025). More broadly, the term is also useful as a shorthand for a recurrent design pattern in audio machine learning: combining complementary representations or modalities—such as intermediate and final acoustic features, audio and video, or speech spectra and listener profiles—through explicit fusion mechanisms rather than relying on a single representation alone (Sun et al., 2022). In that broader sense, AudioFuse refers less to one universal algorithm than to a class of architectures organized around representational complementarity, controlled fusion, and task-specific decoding.

1. Definition and scope

In its most specific usage, AudioFuse is the model introduced in "AudioFuse: Unified Spectral-Temporal Learning via a Hybrid ViT-1D CNN Architecture for Robust Phonocardiogram Classification" (Siddiqui et al., 27 Sep 2025). That model is a lightweight dual-branch deep network for phonocardiogram (PCG) classification that processes a 2D log-Mel spectrogram with a custom Vision Transformer (ViT) and the raw 1D waveform with a shallow 1D CNN, then combines the resulting feature vectors with late fusion. The central claim is that biomedical audio such as PCG is inherently both spectral and temporal/rhythmic, so a model that only sees a spectrogram or only sees a waveform is intrinsically incomplete (Siddiqui et al., 27 Sep 2025).

The same underlying logic appears in other audio research, although often under different names. In automated audio captioning, the Low- and High-Dimensional Feature Fusion (LHDFF) model argues that intermediate CNN activations may retain complementary acoustic detail that is useful for caption generation, and therefore fuses an intermediate “low-dimensional” representation with the final “high-dimensional” representation of a PANNs/CNN10 encoder (Sun et al., 2022). In audio-visual event recognition, MAFnet combines intermediate cross-modal conditioning through a FiLM-based lateral connection with high-level modality-temporal attention, thereby performing fusion at more than one representational level (Brousmiche et al., 2021). In personalized speech enhancement for hearing aids, AFN-HearNet performs middle fusion of encoded speech spectra and an audiogram-derived latent representation through affine modulation inside frequency-temporal Conformer blocks (Ni et al., 9 Sep 2025).

This suggests that “AudioFuse” can be understood in two senses. First, it is the proper name of a specific PCG classifier (Siddiqui et al., 27 Sep 2025). Second, it is a plausible umbrella label for architectures whose core inductive bias is that useful audio information is distributed across heterogeneous but complementary representations, and that explicit fusion is therefore preferable to single-stream processing (Sun et al., 2022).

2. Representational complementarity as the core principle

The explicit motivation of AudioFuse for PCG classification is representational complementarity. A raw discrete audio signal preserves precise event timing, waveform morphology, phase information, and local rhythmic patterns, whereas a spectrogram provides explicit time-frequency organization and strong visibility of tonal and harmonic patterns (Siddiqui et al., 27 Sep 2025). The paper frames waveform and spectrogram as complementary rather than competing representations, because each exposes information the other makes less accessible.

Within this argument, the spectrogram branch is useful because it makes frequency structure directly visible, but it also imposes the usual STFT trade-off and, as stated in the paper, “discards all phase information” and “blurs temporal details” (Siddiqui et al., 27 Sep 2025). The waveform branch preserves exact timing and phase, but requires the model to infer frequency structure indirectly. AudioFuse addresses this mismatch by jointly modeling both representations rather than privileging one.

A closely related argument appears in AAC. The LHDFF model states that most strong AAC systems use a pretrained audio encoder such as PANNs/CNN10 and then feed only its final, high-dimensional representation into a language decoder, but this may discard useful intermediate information (Sun et al., 2022). The proposed solution is to retain and fuse the output of the third convolutional block with the final-layer output, on the premise that intermediate CNN activations may preserve complementary acoustic detail that final abstract embeddings suppress (Sun et al., 2022).

The same principle extends beyond single-modality multi-representation fusion. In MAFnet, the relevance of each modality can change over time within a video, so the model should dynamically answer both which time window matters most and which modality matters most at that time (Brousmiche et al., 2021). In AFN-HearNet, the two inputs are not two views of the same signal but a dynamic speech spectrum and a static sparse audiogram; fusion is motivated by the claim that direct concatenation is inadequate because the two inputs are misaligned in both resolution and structure (Ni et al., 9 Sep 2025). Across these cases, fusion is justified by complementarity, mismatch, or both.

3. Architecture of AudioFuse for phonocardiogram classification

AudioFuse for PCG classification consists of two synchronized input pathways derived from the same recording (Siddiqui et al., 27 Sep 2025). One pathway constructs a 2D log-Mel spectrogram of size $224 \times 224$ and processes it with a custom wide-and-shallow Vision Transformer. The other takes the raw 1D waveform, padded or truncated to a fixed length of 5 seconds, corresponding to 110,250 samples, and processes it with a shallow 1D CNN (Siddiqui et al., 27 Sep 2025).

The spectrogram branch divides the $224 \times 224$ input into 196 patch embeddings, corresponding to a $14 \times 14$ grid and therefore a $16 \times 16$ patch size. Patch extraction is implemented via a Conv2D layer with kernel size $16 \times 16$ , and the embedding dimension is 192. The branch adds learnable positional embeddings, then applies 6 Transformer encoder blocks with 8 attention heads and an MLP, and finally globally averages the output token sequence to obtain a spectral feature vector $\mathbf{f}_{spec} \in \mathbb{R}^{192}$ (Siddiqui et al., 27 Sep 2025).

The waveform branch consists of three sequential Conv1D blocks, each using kernel size 16 and stride 4, followed by Batch Normalization and MaxPooling. The number of filters increases from 64 to 128 to 256 across the three blocks. After the final convolution block, the feature sequence is globally averaged and passed through a final Dense layer with 64 units, yielding $\mathbf{f}_{wave} \in \mathbb{R}^{64}$ (Siddiqui et al., 27 Sep 2025).

Fusion is performed by simple late concatenation: $\mathbf{f}_{fused} = [\mathbf{f}_{spec}; \mathbf{f}_{wave}]$ with $\mathbf{f}_{fused} \in \mathbb{R}^{256}$ (Siddiqui et al., 27 Sep 2025). The fused vector is passed through a small MLP head consisting of Dense(192), Dropout(0.5), and a final sigmoid output layer (Siddiqui et al., 27 Sep 2025). The paper also studies Cross-Attention, Gated FiLM, and Spectrogram-Scalogram fusion, but the core AudioFuse design emphasized in the architecture section is concatenation-based late fusion, which also performs best in the reported comparison (Siddiqui et al., 27 Sep 2025).

For AudioFuse, the spectrogram branch is grounded in the Short-Time Fourier Transform: $X(t, \omega) = \sum_{n=-\infty}^{\infty} x[n] w[n-t] e^{-j\omega n}$ with the power spectrogram defined as

$224 \times 224$ 0

before mapping to the Mel scale (Siddiqui et al., 27 Sep 2025). At the representation level, fusion is expressed directly as concatenation of the branch embeddings: $224 \times 224$ 1 which preserves branch specialization up to a late decision stage (Siddiqui et al., 27 Sep 2025).

The broader AudioFuse pattern includes several distinct fusion operators. In LHDFF for AAC, the encoder output from the third convolutional block is treated as the “low-dimensional” feature $224 \times 224$ 2, while the final-layer output is the “high-dimensional” feature $224 \times 224$ 3 (Sun et al., 2022). Each is projected to 128 dimensions with ReLU: $224 \times 224$ 4

$224 \times 224$ 5

$224 \times 224$ 6

Because $224 \times 224$ 7, the authors state that “the method of patch 0 is used to make the dimension of $224 \times 224$ 8 and $224 \times 224$ 9 the same”, i.e., zero-padding along the temporal axis before fusion (Sun et al., 2022). The text does not define $14 \times 14$ 0 precisely, but the reported output shape implies a same-shape merge rather than concatenation.

On the decoder side, LHDFF uses two Transformer decoders, one conditioned on $14 \times 14$ 1 and one on $14 \times 14$ 2, and fuses the resulting token distributions: $14 \times 14$ 3 Again, $14 \times 14$ 4 is not explicitly defined, but the figure caption states that “the final fused probability result is obtained by fusing the log softmax probability distribution of each transformer decoder” (Sun et al., 2022). This locates fusion at the token-distribution level rather than the feature level.

In MAFnet, fusion is explicitly multi-level. At the intermediate level, visual features modulate audio feature maps through FiLM: $14 \times 14$ 5 where $14 \times 14$ 6 and $14 \times 14$ 7 are generated from the visual representation (Brousmiche et al., 2021). At the high level, the model computes joint modality-temporal attention over modality-time pairs: $14 \times 14$ 8 and forms the global fused representation through weighted temporal pooling within each modality followed by concatenation across modalities (Brousmiche et al., 2021).

In AFN-HearNet, fusion takes the form of affine modulation in latent space rather than concatenation or cross-attention. The encoded speech feature is modulated by audiogram-derived parameters: $14 \times 14$ 9 followed by a second modulation and MLP step: $16 \times 16$ 0 The paper characterizes this as middle fusion, feature-wise conditional modulation, and FiLM-like affine modulation plus gating (Ni et al., 9 Sep 2025).

5. Empirical performance and comparative results

On the PhysioNet 2016 dataset, the PCG AudioFuse model reports ROC-AUC $16 \times 16$ 1 when trained from scratch, compared with 0.8066 for the spectrogram-only baseline and 0.8223 for the waveform-only baseline (Siddiqui et al., 27 Sep 2025). In the same table, AudioFuse (Concatenation) reports Accuracy $16 \times 16$ 2, F1 $16 \times 16$ 3, ROC-AUC $16 \times 16$ 4, and MCC $16 \times 16$ 5 with 2.56M parameters (Siddiqui et al., 27 Sep 2025). The paper also reports AudioFuse (Cross-Attention) at ROC-AUC $16 \times 16$ 6 and AudioFuse (Gated FiLM) at ROC-AUC $16 \times 16$ 7, establishing that the simplest fusion operator performs best in this setting (Siddiqui et al., 27 Sep 2025).

For out-of-domain evaluation on PASCAL Set B, AudioFuse reports Accuracy 0.7386, F1 0.6667, ROC-AUC 0.7181, and MCC 0.4519, compared with ROC-AUC 0.4873 for the spectrogram-only ViT baseline and 0.6782 for the waveform-only 1D CNN baseline (Siddiqui et al., 27 Sep 2025). The paper interprets this as evidence that fusion improves resilience to domain shift and that temporal features are more robust than spectrogram-only features under this particular shift (Siddiqui et al., 27 Sep 2025).

Comparable but task-specific gains are reported in AAC. On Clotho, the baseline without reinforcement learning reports SPIDEr 0.256, while LHDFF reaches 0.265; on AudioCaps, the baseline SPIDEr is 0.404, Pre-BERT gives 0.410, AT-CNN10 gives 0.414, and LHDFF reaches 0.426 (Sun et al., 2022). The ablations are especially important: “LHDFF (only fusi-fea)” reaches SPIDEr 0.261 on Clotho and 0.419 on AudioCaps, supporting the claim that the intermediate low-dimensional feature contributes complementary information (Sun et al., 2022).

In audio-visual event recognition, MAFnet reports 90.86 accuracy on AVE, 86.72 on UCF51, and 83.94 on Kinetics-Sounds (Brousmiche et al., 2021). Its ablation on AVE shows concatenation: 89.34, outperforming addition: 84.77, MCB: 88.83, and DMR: 87.56, while the best conditioning direction is visual $16 \times 16$ 8 audio, yielding 90.86 (Brousmiche et al., 2021). In personalized speech enhancement, AFN-HearNet reports on the DNS-Challenge test set: 1.28M parameters, 3.39 G FLOPs/s, HASQI 0.81, WB-PESQ 2.75, NB-PESQ 3.29, SDR 13.86, SI-SNR 13.62, and STOI 95.55; the paper states that AFN-HearNet significantly outperforms state-of-the-art in-context fusion joint models regarding key metrics such as HASQI and PESQ (Ni et al., 9 Sep 2025).

These results do not establish a single universal superiority of one fusion operator. Instead, they show that the empirical value of AudioFuse-style design depends on the task, the representational mismatch being addressed, and the location at which fusion occurs.

6. Design patterns, limitations, and interpretation

Several recurring design patterns emerge across AudioFuse-related systems. One is late fusion by concatenation, which is the main mechanism in the PCG AudioFuse model and also the strongest fusion operator in MAFnet’s AVE ablation (Siddiqui et al., 27 Sep 2025). Another is intermediate or middle fusion, as in LHDFF’s combination of intermediate and final CNN features and AFN-HearNet’s affine modulation of encoded speech features by an encoded audiogram (Sun et al., 2022). A third is distribution-level fusion, as in LHDFF’s merging of log-softmax outputs from two Transformer decoders (Sun et al., 2022).

A second recurring pattern is that explicit fusion is often paired with architectural restraint. AudioFuse for PCG classification uses a deliberately wide-and-shallow ViT, a shallow 1D CNN, late fusion, dropout 0.5, and early stopping, with a total parameter count of 2.56M for the concatenation model (Siddiqui et al., 27 Sep 2025). The paper argues that these choices mitigate the overfitting risk common in fusion models. Likewise, AFN-HearNet discusses efficiency and hearing-aid suitability explicitly, reporting a theoretical latency of 16 ms under causal constraints and noting the possibility of replacing the fusion module with lighter sequence modules (Ni et al., 9 Sep 2025).

The literature also makes clear that fusion is not identical to general multimodality. LHDFF is explicitly intra-audio, multi-level feature fusion, not cross-modal audio-text fusion inside the encoder (Sun et al., 2022). AFN-HearNet is an audio fusion model in a specialized sense: it fuses a dynamic speech spectrum with a static hearing-loss profile, not audio with video or text (Ni et al., 9 Sep 2025). MAFnet, by contrast, is genuinely audiovisual, but still structured around the idea that fusion should happen at more than one level (Brousmiche et al., 2021).

The limitations are equally consistent. The PCG AudioFuse paper does not report many implementation specifics for spectrogram generation or some branch internals, which limits exact reproducibility from the text alone (Siddiqui et al., 27 Sep 2025). LHDFF leaves several details ambiguous, including the exact meaning of $16 \times 16$ 9, the number of Transformer layers, positional encoding details, and whether PANNs is pretrained and frozen or finetuned (Sun et al., 2022). AFN-HearNet is more explicit about the fusion mechanism, but remains specialized to hearing-aid speech enhancement rather than a generic audio fusion setting (Ni et al., 9 Sep 2025).

This suggests that “AudioFuse” is best interpreted not as a settled canonical architecture, but as a family of explicit fusion strategies whose success depends on careful matching between the representational split and the downstream task.

The significance of AudioFuse-style work lies in the claim that architectural bias can substitute, at least partly, for scale or pretraining when the task structure is well understood. The PCG AudioFuse paper emphasizes that the model is trained from scratch and does not rely on large-scale pretraining, yet remains competitive with substantially larger spectrogram models while being much smaller than DenseNet169 and InceptionV3 (Siddiqui et al., 27 Sep 2025). The practical message is that, for rhythmic biomedical audio, the joint use of spectral and temporal views can provide a strong inductive bias.

A related but more data-centric direction appears in FusionAudio-1.2M, which uses a multimodal contextual fusion pipeline to generate 1.2M detailed captions and 6M QA pairs from AudioSet-derived clips (Chen et al., 1 Jun 2025). Fusion there is not a learned dual-branch classifier but a two-stage automated pipeline that extracts expert cues from multiple modalities—speech, music, general sounds, and visual context—and synthesizes them with an LLM into detailed audio-only captions (Chen et al., 1 Jun 2025). The paper reports that more than 50% of samples integrate two or more modalities, and that removing video, speech, music, or filtering reduces downstream performance (Chen et al., 1 Jun 2025). This indicates that fusion can also function at the level of data construction and supervision rather than only within the final task model.

Another extension is modular infrastructure for hybrid audio systems. NablAFx is presented as an open-source PyTorch framework for differentiable audio effect modeling, supporting black-box and gray-box models, conditioning methods such as FiLM, TFiLM, TTFiLM, TVCond, and TVFiLM, and differentiable DSP blocks (Comunità et al., 17 Feb 2025). Although it is not an AudioFuse paper, it is highly relevant as a substrate for systems that would combine learned waveform processors, controller-mediated conditioning, and explicit DSP modules (Comunità et al., 17 Feb 2025). The paper explicitly notes that hybrid models combining black-box and gray-box processors remain future work, which is a plausible direction for more general AudioFuse-like systems (Comunità et al., 17 Feb 2025).

A further frontier is the unification of heterogeneous front-end tasks in conversational audio systems. UAF reformulates VAD, turn-taking detection, speaker recognition, ASR, and QA as a single auto-regressive sequence prediction problem over semantic and system-control tokens (Li et al., 21 Apr 2026). Although not labeled AudioFuse, it is directly analogous in that it replaces multiple front-end modules with one shared representation and specialized lightweight heads (Li et al., 21 Apr 2026). This suggests that the logic of AudioFuse may extend from representational fusion within a signal to functional fusion across tasks.

Taken together, these works indicate that AudioFuse is both a specific PCG architecture and a more general research orientation. In the narrow sense, it denotes a hybrid ViT–1D CNN classifier for heart sounds (Siddiqui et al., 27 Sep 2025). In the broader sense, it names a design philosophy: preserve complementary information, align heterogeneous representations carefully, and fuse them at a level appropriate to the task rather than assuming that one representation, one modality, or one decoder branch is sufficient (Sun et al., 2022).