--- title: 'HeartMuLa: Music & Cardiac Audio AI' url: https://www.emergentmind.com/topics/heartmula type: topic --- # HeartMuLa: Music & Cardiac Audio AI HeartMuLa is a term used for two distinct advanced systems in contemporary machine learning research: (1) a large-scale, open-source foundation model for controllable music generation using hierarchical large language models and ultra-low-frame-rate tokenization, and (2) a family of state-of-the-art multi-label learning systems for comprehensive analysis of heart and cardiopulmonary sounds, usually framed as murmur attribute extraction and signal separation/classification. Both domains share a commitment to semantic-level audio representation, robust multi-attribute modeling, and hierarchical control of content. Below is a comprehensive account of the methodologies, core architectures, datasets, evaluation protocols, and benchmarks that underpin HeartMuLa research in automatic auscultation and music AI. ## 1. HeartMuLa in Music AI: Hierarchical Foundation Models The HeartMuLa music system [2601.10547] is a hierarchical, LLM-driven framework for music understanding and synthesis, introducing a scalable approach that rivals commercial systems in both fidelity and controllability. ### 1.1 Model Architecture The system comprises four principal components: HeartCLAP (audio-text alignment), HeartTranscriptor (lyric recognition), HeartCodec (an ultra-low-frame-rate RVQ-based music tokenizer), and HeartMuLa proper (the controllable song-generation LLM). - **HeartCodec** compresses 48 kHz stereo music via multi-encoder fusion (Whisper, WavLM, MuEncoder) to feature streams, compresses via a query-based Transformer to 12.5 Hz frame rate, and applies K=8 codebooks with V=8 192 entries for RVQ, yielding a 3 840× compression. - **Generation** is factorized into a global Transformer, $θ_{\rm glo}$, for coarse structure (autoregressively on $a_{l,0}$ tokens), and a local Transformer, $θ_{\rm loc}$, for intra-frame detail (predicting $a_{l,k}$ for $k=1...K-1$). - **Model sizes** include a 3.3B-parameter baseline (3B global + 0.3B local) and a 7.3B-parameter enhanced model, each with ~32–48 layers and 4096 hidden units. ### 1.2 Conditioning and Controllability - **Section-level control** is achieved by interleaving style annotations (tagged by a multimodal LLM) within lyric tokens and using explicit section markers ([intro], [verse], etc.). - **Short-music mode** is deployed via specialized training and inference (denser CFG dropout, shorter context). ### 1.3 Training Protocol Training proceeds in four stages: warmup (30 s clips with lyric/audio condition), full-dataset pretraining, supervised fine-tuning (high-quality subset, increased loss weighting), and direct preference optimization (DPO) with RL-style feedback from MuQ, AudioBox, SongEval, and PER datasets. ## 2. HeartMuLa in Biomedical Audio: Multi-Attribute Heart Sound Analysis HeartMuLa is also the term for a family of deep-learning approaches targeting comprehensive heart-sound analysis, especially multi-label murmur attribute prediction, as pioneered in Deep CardioSound [2204.07420], and audio-LLM transfer [2501.13884]. ### 2.1 Problem Definition The core task extends beyond binary murmur detection to automatic labeling of phonocardiogram (PCG) segments with multiple clinically-used attributes: timing, grading, pitch, quality, and shape [2204.07420]. The analysis typically focuses on the systolic period; diastolic analysis is less commonly addressed due to data scarcity. ### 2.2 Datasets and Annotation - **CirCor DigiScope**: 5,272 PCG recordings from 1,577 pediatric patients, annotated at the patient/segment level for up to five murmur groups and detailed cardiac segment boundaries (S1, systole, etc.) [2204.07420, 2501.13884]. - **Manikin-Recorded Cardiopulmonary Sounds**: 210 high-fidelity, filter-enhanced heart and lung audio clips (separated and mixed), annotated for 10 heart and 6 lung conditions, with location metadata [2410.03280]. ## 3. Deep Learning Architectures and Audio Representation ### 3.1 Multi-Branch CNN Ensembles Deep CardioSound employs a five-branch CNN ensemble architecture: each branch specializes in one murmur attribute group and shares a DenseNet-121 as frontend, operating on "bags" of consecutive, preprocessed systolic segments. Feature fusion and a global consistency branch augment the multi-label structure. Cross-entropy losses are combined across groups, with no explicit focal/imbalance weighting; data augmentation is used to mitigate label skew [2204.07420]. ### 3.2 Audio-LLM Approaches Recent advances involve fine-tuning large-scale audio LLMs such as Qwen2-Audio [2501.13884]. These use Whisper-style audio encoders (SSM transformer hybrids) to produce embeddings, which are then processed by LLMs (e.g., Qwen-7B) for multitask classification (timing, grading, pitch, etc.), with LoRA adapters for parameter-efficient training. Segmentation with SSAMBA is used to feed phase-specific representations (S1, S2, etc.) to the model. ### 3.3 Preprocessing and Feature Extraction Approaches utilize denoising (Butterworth filters), precise segmentation (via HSMM or self-supervised models), normalization, and time-frequency representations (MFCC, STFT, Wavelet Scattering Transform). The blending of fine-grained signal processing with hierarchical deep learning distinguishes the state of the art [2303.11423, 2410.03280]. ## 4. Evaluation Protocols and Benchmarks ### 4.1 Music Model Evaluation - **Objective metrics**: SongEval, AudioBox, Tag-Sim, PER (lyric intelligibility). - **Subjective metrics**: Mean Opinion Scores (MOS) for musicality, structure, text alignment. - **Performance**: HeartMuLa-7B achieves competitive or superior scores on PER and subjective musicality relative to leading commercial models such as Suno-v5 and MiniMax-2.0. Batch inference with optimized attention reduces end-to-end song generation times by 5.4× [2601.10547]. ### 4.2 Biomedical Audio Evaluation - **Segment-level**: Sensitivity, specificity, F1, precision. - **Patient-level**: Label group mode aggregation, overall accuracy (up to 96.9%) [2204.07420]. - **Feature-specific**: Multi-class accuracy and weighted-accuracy (W.acc) for 11 murmur tasks (timing, grading, pitch, quality, shape—systolic and diastolic), with audio-LLM methods achieving 100% on most, except "grading" where text encoder limitations are observed [2501.13884]. - **Comparisons**: Deep CardioSound and audio-LLM models surpass prior DNNs in multi-label F1 and classification of long-tail attributes. | Model / Task | Sys. Timing | Sys. Shape | Sys. Grading | W.acc (murmur) | |----------------------|-------------|------------|--------------|----------------| | Deep CardioSound | 96.6% | 96.3% | 96.6% | — | | Qwen2-Audio (with seg.) | 100% | 100% | 33.4% | 75.6% | | M2D+AST | — | — | — | 83.2% | ## 5. Signal Processing, Separation, and Physiology Modeling ### 5.1 Heart-Lung Source Separation Using the manikin-mixed dataset, techniques such as affine non-negative matrix factorization (ANMF), deep U-Nets, and multimodal self-supervised encoders enable monaural separation of overlapped heart and lung sounds, with SNR gains of 6–10 dB reported for ANMF [2410.03280]. The dataset is designed to benchmark "HeartMuLa"-style front-ends. ### 5.2 Multiscale Hemodynamics The term "HeartMuLa" also appears as the label for a geometric multiscale simulation platform: 3D Navier–Stokes in an ALE framework with RIIS valve models, one-way kinematic coupling from 3D electromechanics, and 0D closed-loop circuit for the rest of the circulation [2110.02114]. Physiological outputs—LV volume, ejection fraction, pressure flow—are validated against gold-standard MRI and echo data. ## 6. Limitations, Challenges, and Future Directions - **Segment-label reliance**: Most biomedical HeartMuLa models require high-quality S1–S2 segmentation or prior knowledge of class distributions; errors in segmentation or rare label groups (especially diastolic murmurs and "grading") propagate directly into final outputs [2204.07420, 2501.13884]. - **Domain adaptation**: Transfer learning across populations and device modalities remains unresolved for heart sound models [2303.11423]. - **Generalization**: While music HeartMuLa scales well with model size and data, open questions remain in multi-lingual lyric alignment and style transfer [2601.10547]. - **Data limitations**: Simulator-only datasets lack patient variability and natural artifact, calling for augmentation with real-world recordings [2410.03280]. - **Next steps**: Joint modeling of heart sounds with ECG/imaging, multimodal LLMs, uncertainty quantification, improved text encoders, and end-to-end learning from waveform to structured diagnosis are identified as priorities for future research [2501.13884]. ## 7. Impact and Practical Implications HeartMuLa systems—both musical and biomedical—have significantly advanced the capacity for large-scale, controllable generation and multi-label analysis of complex audio. In clinical contexts, HeartMuLa-based models enable structured, explainable murmur assessments spanning multiple attributes, supporting differential diagnosis and potentially automating auscultation and triage. In music AI, HeartMuLa demonstrates that academic-scale resources can now approach or exceed commercial-grade song synthesis in flexibility, fidelity, and interpretability. The convergence of semantic-token audio representations, hierarchical generative modeling, and multi-condition control represents a paradigm shift in both AI-driven audio understanding and production [2204.07420, 2601.10547, 2501.13884]. Source: https://www.emergentmind.com/topics/heartmula