Cryfish: Deep Audio Analysis LLM

• Cryfish is an audio-language model that integrates a universal audio encoder, transformer connector, and LLM backend to perform diverse auditory tasks.
• It overcomes challenges of context length and task diversity by compressing audio features into instruction-compatible tokens for heterogeneous processing.
• Cryfish demonstrates state-of-the-art performance on the Dynamic SUPERB benchmark, excelling in tasks like ASR, speaker identification, and audio captioning.

Searching arXiv for Cryfish and closely related AudioLLM papers to ground the article and collect citation IDs. arxiv_search query: "Cryfish On deep audio analysis with LLMs" arxiv_search results:

1. (Mitrofanov et al., 18 Aug 2025) — "Cryfish: On deep audio analysis with LLMs"
2. (Milbradt et al., 2024) — "Audio Flamingo 2: An Audio-LLM with Long-Audio Understanding and Expert Reasoning Abilities"
3. (Miki et al., 3 Feb 2025) — "Think Sound: Speech Chain-of-Thought Reasoning in LLMs" arxiv_search query: "Qwen2-Audio arXiv" arxiv_search results:
4. (Chu et al., 2024) — "Qwen2-Audio Technical Report"
5. (Pal et al., 2024) — "MusiConGen: Refining Diffusion Music Generation with Minimal and Generic Text Attributes"
6. (Micheli, 2024) — "Qwen2.5-Omni Technical Report" arxiv_search query: "SALMONN speech audio LLM arXiv" arxiv_search results:
7. (Tang et al., 2023) — "SALMONN: Towards Generic Hearing Abilities for LLMs"
8. (Melzer et al., 29 Apr 2025) — "Is It Really Better? Understanding the Evaluation of Audio Captioning"
9. (Zhang et al., 2024) — "Multi-modal Situated Reasoning in Virtual Reality" arxiv_search query: "WavLM self-supervised speech audio representation model arXiv" arxiv_search results:
10. (Chen et al., 2021) — "WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing"
11. (Addazi et al., 2 May 2025) — "Questioning the value of self-supervised speech models for speech information retrieval"
12. (Huang et al., 2022) — "SpeechLM: Enhanced Speech Pre-Training with Unpaired Textual Data" arxiv_search query: "\"Dynamic SUPERB Phase-2\" arXiv" arxiv_search results:
13. (Adlam, 2024) — "Dynamic-SUPERB Phase-2: A Collaborative, Expanding, and Dynamic Benchmark for Spoken LLMs"
14. (Addazi et al., 2 May 2025) — "Questioning the value of self-supervised speech models for speech information retrieval"
15. (Olson et al., 2024) — "MLLMs Know Where to Look: Training-Free Perception of Small Visual Details with Multimodal LLMs" arxiv_search query: "\"Whisper-LLaMA\" arXiv" arxiv_search results:
16. (Gorbunov et al., 2024) — "Whispering LLaMA: A Cross-Modal Generative Error Correction Framework for Speech Recognition"
17. (Huffman et al., 2024) — "MaLa-ASR: Advancing low-resource automatic speech recognition with largescale LLMs"
18. (Esterov et al., 2024) — "STT-LLM: Bridging Speech-to-Text and LLMs" arxiv_search query: "\"LoRA: Low-Rank Adaptation\" arXiv" arxiv_search results:
19. (Hu et al., 2021) — "LoRA: Low-Rank Adaptation of LLMs"
20. (Parthasarathy et al., 2024) — "LoRA Rank Growth: Breaking the Trade-off Between Efficiency and Performance"
21. (Otter et al., 2024) — "Federated Fine-Tuning of LLMs under Heterogeneous Language Tasks and Parameter-Efficient Training Methods" arxiv_search query: "\"WavLLM\" arXiv" arxiv_search results:
22. (Rodrigues et al., 2024) — "WavLLM: Towards Robust and Adaptive Speech LLM"
23. (Sotirov et al., 24 Jun 2025) — "M2M-Whisper: Translating from Millions of Voices"
24. (Gorbunov et al., 2024) — "Whispering LLaMA: A Cross-Modal Generative Error Correction Framework for Speech Recognition" Cryfish is an audio-capable LLM designed for deep audio analysis across speech recognition, language and speaker identification, noise and distance estimation, audio captioning, music understanding, environmental sound understanding, and related auditory tasks. Introduced in "Cryfish: On deep audio analysis with LLMs" (Mitrofanov et al., 18 Aug 2025), it belongs to the recent line of AudioLLMs that extend text LLMs toward multimodal perception, alongside systems such as SALMONN (Tang et al., 2023), Qwen2-Audio (Chu et al., 2024), and WavLLM (Rodrigues et al., 2024). Its stated objective is not merely to add automatic speech recognition, but to provide broad listening abilities that generalize across heterogeneous auditory domains.

1. Conceptual scope and problem setting

Cryfish is framed around a specific difficulty in multimodal LLM design: integrating listening capabilities into an LLM so that the model can handle complex auditory tasks across speech and sounds rather than a narrow speech-centric subset (Mitrofanov et al., 18 Aug 2025). The paper identifies four obstacles. First, audio tasks are diverse: speech recognition, speaker verification, language identification, emotion recognition, sound events, music, and spatial audio require different combinations of linguistic, paralinguistic, spectral, temporal, and spatial information. Second, common audio encoders can be overly specialized; the paper contrasts self-supervised and supervised pretraining regimes and notes that encoders optimized for ASR may be less suited for general audio events or subtle prosody. Third, raw audio creates a context-length problem because it yields thousands of frames, making direct ingestion by a transformer LLM impractical. Fourth, the model must follow natural-language instructions while remaining grounded in the signal rather than relying on textual priors or dataset metadata.

Within that setting, Cryfish is presented as a general-purpose AudioLLM rather than a task-specific recognizer. The emphasis on generalization is central: the model is evaluated on Dynamic SUPERB Phase-2, a broad multitask benchmark intended for spoken LLMs and AudioLLMs rather than a narrowly speech-only suite (Adlam, 2024). This positioning distinguishes Cryfish from systems optimized for a single auditory capability and explains why the paper repeatedly compares it both to open AudioLLMs and to dedicated specialist models.

2. Architecture and signal-to-language interface

Cryfish combines three components: a WavLM audio encoder, a transformer-based connector, and a Qwen2.5-7B-Instruct language backbone adapted with LoRA (Chen et al., 2021, Hu et al., 2021). The paper’s architectural claim is that a single universal encoder plus a trainable connector can supply both global and temporal auditory information to the LLM without the instability observed in a multi-encoder mixture.

WavLM serves as the universal encoder. Given a raw waveform $x$ sampled at 16 kHz, it produces frame-level embeddings

$$\mathbf{H}_{\text{wav}} = \text{WavLM}(x) \in \mathbb{R}^{T \times d_{\text{wav}}}.$$

In Cryfish, WavLM is frozen during most of training and only unfrozen in the last epoch. The paper states that WavLM plus the connector together comprise 345M parameters, indicating that the audio front end is a substantial representational backbone rather than a lightweight preprocessor.

The connector is a pure transformer connector rather than a Q-Former. Structurally, it projects WavLM features into a connector space, processes them through transformer layers, and compresses them into two output forms: 5 sentence-level embeddings and a sequence of frame-level embeddings at 2.5 Hz (Mitrofanov et al., 18 Aug 2025). The resulting audio token sequence is

$$\mathbf{A} = [\mathbf{s}_1, \dots, \mathbf{s}_5, \mathbf{f}_1, \dots, \mathbf{f}_M] \in \mathbb{R}^{(5 + M) \times d_c}.$$

This design preserves a global summary for tasks such as captioning or overall classification while retaining coarse temporal structure for tasks such as event detection, distance prediction, or editing detection.

The LLM backbone is Qwen2.5-7B-Instruct, adapted with a 21M-parameter LoRA module. Audio tokens are projected into the LLM token space and concatenated with the text instruction before autoregressive generation. The training objective is the standard cross-entropy loss over generated tokens:

$$\mathcal{L}_{\text{CE}} = -\sum_t \log p_\theta(y_t \mid y_{<t}, \mathbf{X}).$$

The architecture therefore implements a compressed audio-to-token interface rather than direct frame-level conditioning at LLM scale. This suggests that Cryfish’s core technical contribution lies less in the language backbone itself than in the design of the audio representation bottleneck.

3. Training regime and instruction construction

Cryfish uses a two-stage training strategy (Mitrofanov et al., 18 Aug 2025). Stage 1 is template-based instruction training, intended to establish baseline audio understanding on core supervised tasks. It uses approximately 3.2M template-based instructions across 7 tasks: ASR, gender recognition, speaker identification or verification, language identification, noise classification, distance prediction, and SNR prediction. The listed datasets include Auto-ACD, WavCaps, MLS, FLEURS, CHiME-6, Common Voice, augmented LibriSpeech with RIRs and noise, VoxCeleb2, and phoneme text databases. Training runs for two epochs: in the first epoch WavLM is frozen while the connector and LoRA are trained; in the second epoch WavLM is unfrozen. The optimizer is Adam, the scheduler is OneCycleLR with max LR $= 1 \times 10^{-3}$, and the dynamic batch size is approximately 3000 tokens per batch, counting both text and audio tokens inside the LLM context.

Stage 2 is LLM-generated instruction training, intended to improve instruction-following and generalization beyond rigid templates. It uses approximately 0.8M instructions, of which 0.6M are LLM-generated and 0.2M are additional template-based instructions. The paper lists 44 datasets spanning speech, audio events, music, anti-spoofing and safety, emotion, prosody and accent, health and anomaly detection, and bird calls. The generation procedure combines dataset descriptions and metadata tags into prompts for a text LLM, specifically Qwen2.5, which then produces natural-language question-answer pairs consistent in style with Qwen inside Cryfish.

The paper reports quality-control issues in this process, including bias toward yes/no questions and overuse of metadata rather than genuine audio content. These were mitigated with regex-based filtering and fallback to simpler templates. A manual evaluation of 250 random instructions yielded ~0.795 accuracy, which the authors considered acceptable (Mitrofanov et al., 18 Aug 2025). This training design is consequential because it makes instruction diversity itself part of the model’s auditory competence: Cryfish is not only trained on audio labels, but on mappings between audio and heterogeneous instruction styles.

4. Benchmark design and evaluation methodology

Cryfish is evaluated primarily on Dynamic SUPERB Phase-2, described as a collaboratively expanding benchmark for spoken LLMs and AudioLLMs (Adlam, 2024). The full benchmark covers 180 tasks, while the Cryfish evaluation uses 154 tasks. These tasks span ASR, speech-to-text translation, speaker verification and diarization, language identification, prosody and emotion tasks, audio spatial distance prediction, music understanding, audio event detection, and captioning, with approximately 200 examples per task.

The evaluation protocol mixes direct metric computation and LLM-as-a-judge scoring. For classification tasks, Cryfish outputs free-form text, and a judge LLM maps that output to correctness. The paper states that several judges were tested and that LLaMA 3.3 70B achieved 97.78% evaluation accuracy on a balanced human-annotated set, corresponding to 2468/2524 correct (Mitrofanov et al., 18 Aug 2025). The classification score is reported as

$$\text{LLM-C} = \frac{\# \text{ correct answers}}{\# \text{ total samples}} \times 100\%.$$

Regression tasks parse numeric values from text outputs. For spatial distance prediction, the paper uses Median Absolute Error and an NA-adjusted variant that incorporates the Not Applicable Rate:

$$\text{NA-adj medae} = \frac{\text{medae} \downarrow}{1 - \text{NAR} \downarrow}.$$

Sequence generation tasks use BLEU for speech-to-text translation and WER for ASR:

$\text{WER} = \frac{S + D + I}{N} \downarrow.$

Because the benchmark mixes heterogeneous metrics, the paper also computes relative per-task scores normalized with respect to a Whisper-LLaMA baseline and then aggregates domain-level and global performance. The methodological significance is that Cryfish is evaluated as a general auditory reasoning system rather than only as a transcription engine.

5. Empirical results and comparative standing

The paper reports that Cryfish achieves the strongest overall results among the publicly compared open AudioLLMs on Dynamic SUPERB Phase-2 (Mitrofanov et al., 18 Aug 2025). Representative results are summarized below.

Task or aggregate	Metric	Cryfish
SuperbST	BLEU $\uparrow$	35.36
MLS Italian ASR	WER $\downarrow$	29.55
MLS Polish ASR	WER $$\mathbf{H}_{\text{wav}} = \text{WavLM}(x) \in \mathbb{R}^{T \times d_{\text{wav}}}.$$0	46.32
SUPERB ASR	WER $$\mathbf{H}_{\text{wav}} = \text{WavLM}(x) \in \mathbb{R}^{T \times d_{\text{wav}}}.$$1	5.68
Audio Spatial Distance Prediction	NA-adjusted medae $$\mathbf{H}_{\text{wav}} = \text{WavLM}(x) \in \mathbb{R}^{T \times d_{\text{wav}}}.$$2	0.84 (0.76/0.90)
Average LLM-C (Speech)	$$\mathbf{H}_{\text{wav}} = \text{WavLM}(x) \in \mathbb{R}^{T \times d_{\text{wav}}}.$$3	51.57
Average LLM-C (Audio)	$$\mathbf{H}_{\text{wav}} = \text{WavLM}(x) \in \mathbb{R}^{T \times d_{\text{wav}}}.$$4	37.71
Total relative-score-based performance	$$\mathbf{H}_{\text{wav}} = \text{WavLM}(x) \in \mathbb{R}^{T \times d_{\text{wav}}}.$$5	13.85

On specific benchmark rows, Cryfish scores 35.36 BLEU on SuperbST, substantially above Whisper-LLaMA, SALMONN 7B, SALMONN 13B, and Qwen2-Audio; 29.55 WER on MLS Italian ASR and 46.32 WER on MLS Polish ASR, both reported as the best among the compared systems; and 5.68 WER on SUPERB ASR, described as strong and second only to SALMONN 13B. For Audio Spatial Distance Prediction, Cryfish records 0.84 (0.76/0.90) in NA-adjusted medae, with lower being better.

At the domain level, Cryfish reaches 51.57 average LLM-C in speech and 37.71 average LLM-C in audio, both the highest values among the models compared in the paper. Its relative-score-based performance is 10.22 for speech, 22.64 for audio, and 13.85 overall, again the highest among the open AudioLLMs under comparison. Figure-level analysis in the paper states that Cryfish consistently outperforms other AudioLLMs across 19 domains.

Two task-specific analyses clarify both capability and limit. In speaker verification, all Cryfish variants outperform SALMONN-7B, especially on VOiCES, where Cryfish records 29.2 EER versus 50 for SALMONN-7B; however, Cryfish remains far behind the task-specific ECAPA-TDNN baseline, which reaches 4.83 on VOiCES. In language identification, Cryfish reports 66.2 open-ended and 83.4 closed-set accuracy on VoxLingua107, and 67.4 open-ended and 89.3 closed-set accuracy on FLEURS. The paper states that these results significantly exceed those of SALMONN, Qwen-audio, and Qwen2-audio, and that on FLEURS Cryfish beats Whisper in the open setting while approaching MMS and XLS-R in closed-set conditions. This pattern supports the paper’s broader claim: Cryfish does not uniformly surpass specialist systems, but it narrows the gap while maintaining much broader task coverage.

6. Interpretation, limitations, and prospective use

The paper’s analysis attributes Cryfish’s performance to two design choices: the use of a strong self-supervised universal encoder and the transformer-based connector (Mitrofanov et al., 18 Aug 2025). It explicitly reports that a WavLM-based single encoder was more stable and effective than a multi-encoder mixture involving Whisper, BEATs, and ECAPA. It also argues that the connector is central because it compresses and aligns audio features into an LLM-compatible token sequence while preserving both global and temporal cues. This suggests that connector design, not only encoder choice, is a major determinant of general AudioLLM behavior.

The training analysis identifies two important failure modes. One is prompt-class imbalance, where the model can exploit prompt regularities rather than audio content. The other is imbalance between short-answer and long-generation tasks, since short answers naturally yield lower cross-entropy loss and can dominate optimization unless losses are masked or reweighted. These observations are important because they frame AudioLLM training as a problem of multitask loss geometry and instruction distribution, not simply one of scale.

Several limitations are explicitly acknowledged. Cryfish remains weaker than dedicated models on some tasks, notably ECAPA-TDNN in speaker verification and MMS or XLS-R at full performance in language identification. The paper also notes data coverage and instruction-generation biases, prompt sensitivity, latency and training cost, and the possibility that Dynamic SUPERB Phase-2’s approximately 200 examples per task may yield noisy estimates. Future directions include better audio-text alignment, larger and more diverse audio datasets, improved benchmarks, revisiting multi-encoder fusion with better optimization, and extending the framework to joint audio-vision-text models.

The practical applications listed in the paper include audio assistants, sound event monitoring, accessibility tools, multimodal analysis systems, and audio forensics and safety. In that sense, Cryfish is best understood not as a replacement for high-performance specialist recognizers, but as a general-purpose auditory interface for instruction-following LLMs. A plausible implication is that its importance lies in demonstrating that a mid-scale LLM combined with a universal audio encoder and a carefully engineered connector can outperform other open AudioLLMs on broad multitask evaluation while remaining extensible to richer multimodal systems.