TinyMusician: On-device Music Generation

• TinyMusician is a lightweight on-device music generation model that employs advanced knowledge distillation and quantization to reduce model size by 55% while retaining 93% of the teacher's performance.
• It utilizes Stage-mixed Bidirectional and Skewed KL-Divergence for effective training and Adaptive Mixed-Precision Quantization to ensure efficient deployment on resource-constrained devices.
• The model demonstrates high audio fidelity with competitive FAD scores and robust text-audio alignment, enabling practical applications in mobile composition, game audio, and educational tools.

TinyMusician is a lightweight, mobile-deployable music generation model specifically designed to eliminate cloud dependency and enable high-fidelity automatic music creation directly on resource-constrained devices. The model is distilled from MusicGen-Small, a Transformer-based architecture, and incorporates two principal innovations to effectively compress the teacher’s capabilities: (i) Stage-mixed Bidirectional and Skewed Kullback–Leibler (KL) Divergence for advanced knowledge distillation, and (ii) Adaptive Mixed-Precision Quantization for efficient deployment and inference. TinyMusician achieves approximately 93% of the teacher’s performance while reducing model size by 55%, and is deployed fully on-device, setting a new standard for practical music generation accessibility.

1. Model Architecture and Knowledge Distillation

TinyMusician utilizes a transformer backbone transferred via knowledge distillation from MusicGen-Small. The core innovation is the Stage-mixed Bidirectional and Skewed KL-Divergence, which dynamically guides the student model’s learning across different temporal segments of musical structure:

• Loss Formulation:

$$\mathcal{L}_{\mathrm{kl}}(t) = \alpha(t) \cdot [\gamma_1 D_{\mathrm{KL}}(T \| S) + (1-\gamma_1)D_{\mathrm{KL}}(T \| S_{\ell})] + (1-\alpha(t)) \cdot [\gamma_2 D_{\mathrm{KL}}(S \| T) + (1-\gamma_2)D_{\mathrm{KL}}(S \| T_{\ell})]$$

Here, $T$ and $S$ represent teacher and student output distributions, $S_{\ell}$ and $T_{\ell}$ are convex mixtures of the student and teacher, $\gamma_1, \gamma_2$ are balance coefficients, and the dynamic weighting $\alpha(t)$ controls the loss stage via:

$$\alpha(t) = \begin{cases} 1, & t < \tau_{\text{step}} \ 0, & \text{otherwise} \end{cases}$$

This allows early emphasis on global structure (i.e., chronological coherence) and later focus on fine local details (e.g., tonality, rhythm).

• Benefits: This dynamic, bidirectional loss stabilizes convergence, reduces oscillations in training, and mitigates the risk of overfitting to either global or local structures. The resulting student model robustly approximates the complex distribution of the teacher, despite significantly fewer trainable parameters.

2. Adaptive Mixed-Precision Quantization

To achieve device-level efficiency, TinyMusician employs partitioned quantization strategies:

• Component-wise Quantization:
  • Text Encoder (T5): Int8 precision for embedding efficiency, maintaining core sequence semantics.
  • MusicGen-Decoder: Float16 precision for stability in autoregressive music token generation.
  • Encodec-Decoder: Full Float32 precision to retain reconstruction fidelity in audio waveform synthesis.
• Temperature Annealing:

Adaptive temperature decay is implemented via a linear schedule:

$\tau = T_b - (T_b - T_f) \cdot \frac{s}{L_{\max}}$

Where $T_b$ and $T_f$ are begin/end temperatures, $s$ is the current inference step, and $L_{\max}$ is output length.

• Deployment Format:

The entire PyTorch model is converted to ONNX via Optimum-Cli, optimizing for native mixed precision and fast inference.

3. Performance and Evaluation

TinyMusician’s design enables high performance and resource savings:

• Audio Fidelity:
  • Achieves FAD scores of 6.44, closely matching MusicGen-Small’s 6.49.
  • When quantized, the model maintains competitive fidelity with only minor degradations.
• Text-Audio Alignment:
  • CLAP alignment scores improve under mixed-precision deployment (e.g., from 0.303 to 0.373).
• Resource Consumption:
  • Model size reduced from 3.2GB (baseline) to 1.04GB (quantized).
  • Inference times measured at 26.54 seconds (quantized) versus 10 seconds (full precision).
  • Lowered GPU/CPU memory requirements facilitate deployment on edge devices.
• Comparative Results:

The ablation paper demonstrates improved training stability and slightly improved FAD scores with bidirectional KL-distillation. TinyMusician outperforms bulkier models (including YuE-7B and DiffRhythm) in alignment and mobile readiness.

4. Edge Deployment and Practical Applications

TinyMusician is validated in a real-world mobile environment:

• ONNXRuntime Integration:

Runs efficiently and natively on consumer devices (e.g., iPhone 16 Pro, iOS 18.2) using hardware-optimized kernels.

• Interface and User Experience:

Screenshots and system logs in the original work detail the generation workflow: text input, processing, and music preview, executed entirely on-device.

• Application Domains:
  • Mobile composition environments.
  • In-app generative assistants (music, sound effects).
  • Game audio engines with low-latency, real-time adaptive soundtracks.
  • Educational music tools for schools and remote learning.

5. Experimental Analysis and Limitations

Scrutiny of experimental configurations reveals trade-offs:

• Efficiency vs. Fidelity:

Quantization yields substantial reductions in memory and model size, though at the cost of increased latency and, in some configurations, a ~9.5% reduction in melodic/harmonic fidelity.

• Training Dynamics:

Dynamic loss weighting ($\alpha(t)$) results in smoothly descending training curves compared to conventional distillation, improving practical reliability in deployment.

• Limitations:
  • Quantized kernel optimization requires further development to fully leverage hardware for lower latency.
  • Minor reductions in text-to-audio alignment associated with aggressive compression.
  • Future extension to larger, multi-modal music generation models and further exploration of the precision-performance frontier is required.

6. Future Directions

Potential research extensions and refinements are identified:

• Kernel and runtime optimizations to further minimize inference latency in quantized mode.
• Integration with multimodal or cross-domain models (text+image+music) and expansion to larger MusicGen variants.
• Granular quantization schedules and compression strategies to target optimal trade-offs between resource consumption and audio fidelity.
• Broader benchmarking across genres and complex musical tasks, addressing harmonic richness, stylistic nuance, and polyphony.

TinyMusician establishes a device-local paradigm for generative music models, grounded in advanced distillation and quantization principles. Its empirically validated performance and efficient architecture position it as a reference model for embedded, resource-aware creative AI in music generation (Wang et al., 31 Aug 2025).