Resp-Agent: Multimodal Respiratory Diagnostics

• Resp-Agent is an autonomous multimodal system that integrates an LLM-based planner, modality-weaving diagnoser, and flow-matching generator for respiratory analysis.
• The closed-loop design actively addresses transient acoustic events and severe class imbalance to enhance diagnostic accuracy.
• It leverages high-fidelity synthetic data and a benchmark corpus (Resp-229k) to improve disease diagnosis across diverse and rare classes.

Resp-Agent is an autonomous multimodal system for respiratory sound generation and disease diagnosis designed to address two stated limitations of deep learning-based respiratory auscultation: inherent information loss when signals are converted into spectrograms, which discards transient acoustic events and clinical context, and limited data availability under severe class imbalance. The system combines an LLM-based planner, a multimodal diagnoser, and a controllable generator in a closed loop, and is paired with Resp-229k, a benchmark corpus of 229,101 quality-controlled recordings with LLM-distilled clinical narratives (Zhang et al., 16 Feb 2026).

1. Closed-loop multi-agent formulation

Resp-Agent is organized as a three-agent system under the control of an LLM-based planner called Thinker-A$^2$CA, the Active Adversarial Curriculum Agent. In the reported implementation, Thinker-A$^2$CA uses DeepSeek-V3.2-Exp, a 7B-parameter LLM. Its role is to parse a high-level intent such as “improve performance on underdiagnosed classes,” inspect current Diagnoser error profiles including per-class F1, uncertainty, and domain shift, and formulate a “curriculum,” namely a small batch of synthetic generation requests specified as $$\langle \text{disease\_label}, \text{domain} \rangle$$ pairs. The planner recycles model confidences and rationales, calibration scores, and error clusters by class and source domain, with the stated optimization objective of maximizing Macro-F1 on tail classes under a total synthesis budget $B$ (Zhang et al., 16 Feb 2026).

Component	Implementation	Stated role
Thinker-A$^2$CA	DeepSeek-V3.2-Exp, 7B	Inspect weaknesses and schedule targeted synthesis
Generator	Flow Matching Generator	Produce controllable respiratory sounds from disease and style conditions
Diagnoser	Multimodal Longformer	Output diagnostic predictions and calibrated confidences

The closed-loop workflow alternates between diagnosis and targeted synthesis. At each iteration, the Diagnoser is trained on the current union of real and synthetic data, evaluated on a validation split to compute per-class $F1_i$, uncertainty $U_i$, and domain errors, and then queried by Thinker-A$^2$CA for a plan $P=\{(c_j,\text{domain}_j)\}_{j=1..k}$ with $k \leq$ remaining budget. The Generator synthesizes the requested clips and appends them to the synthetic set. This design is explicitly contrasted with static pipelines: the planner is not a one-time sampler, but a central controller that actively identifies diagnostic weaknesses and schedules targeted synthesis in a closed loop (Zhang et al., 16 Feb 2026).

Under a fixed synthesis budget of $^2$0k synthetic samples on the held-out cross-domain test set, the reported Macro-F1 values are 0.212 for the no-synthesis CE-loss baseline, 0.442 for random sampling, 0.512 for class-prior rebalancing, 0.546 for static uncertainty sampling, and 0.598 for Thinker-A$^2$1CA. This suggests that the main contribution of the agentic loop is not merely adding synthetic data, but selecting synthetic data in response to current diagnostic failure modes (Zhang et al., 16 Feb 2026).

2. Modality-weaving diagnoser

The Diagnoser is a multimodal Longformer that fuses textual EHR summaries with raw audio features at the input layer through what is termed a Modality-Weaving Diagnoser. The EHR summary is tokenized into a sequence beginning with [CLS] and ending with [SEP], and $^2$2 placeholder tokens [AUDIO_EMBED] are reserved in the same sequence. From a 10 s, 16 kHz waveform $^2$3, BEATs features are extracted as

$^2$4

aligned by deterministic crop/pad, and projected with a learned matrix $^2$5 to obtain

$^2$6

The 496 placeholders are then replaced with these audio embeddings, while the remaining tokens use standard text embeddings (Zhang et al., 16 Feb 2026).

The attention mechanism is Strategic Global Attention on a Longformer backbone. The complexity is given as $^2$7 for local sliding-window attention and $^2$8 for global attention. The global-token set is defined as

$^2$9

where

$$\langle \text{disease\_label}, \text{domain} \rangle$$0

This places an anchor every 4 frames, approximately 80 ms, so that brief events such as wheezes and crackles can be directly queried by any global token, including the text sentinels. The reported result is sub-100 ms sensitivity to transients with linear memory (Zhang et al., 16 Feb 2026).

The stated architectural benefits are native early fusion at the token level and cross-modal hubs through anchors that enable long-range text-to-audio and audio-to-text routing. Ablations quantify the contribution of these design choices. Removing anchors drops Macro-F1 from 0.212 to 0.189. Removing weaving drops accuracy to 0.650. On the cross-domain test setting, late fusion with raw metadata and no anchors yields Acc = 0.780 and F1 = 0.145; weaving without anchors and with raw metadata yields Acc = 0.640 and F1 = 0.175; full weaving with anchors and LLM-EHR yields Acc = 0.849 and F1 = 0.212 (Zhang et al., 16 Feb 2026).

3. Flow-matching generator and controllable synthesis

The Generator is a two-stage pipeline. In the first stage, Resp-MLLM retools a text-only LLM, Qwen3-0.6B, for discrete-unit modeling without changing the underlying LLM. Modality injection compresses framewise BEATs features $$\langle \text{disease\_label}, \text{domain} \rangle$$1 into $$\langle \text{disease\_label}, \text{domain} \rangle$$2 style descriptors:

$$\langle \text{disease\_label}, \text{domain} \rangle$$3

where StyleProj is a 2-layer MLP. The prompt is [DIAGNOSIS] d [AUDIO_0] ... [AUDIO_{K-1}], with the AUDIO_i embeddings replaced by $$\langle \text{disease\_label}, \text{domain} \rangle$$4. The model autoregressively predicts a sequence of discrete BEATs acoustic units $$\langle \text{disease\_label}, \text{domain} \rangle$$5 from a codebook of size $$\langle \text{disease\_label}, \text{domain} \rangle$$6 by minimizing

$$\langle \text{disease\_label}, \text{domain} \rangle$$7

To obtain leak-free conditioning, approximately 10% of preceding tokens are randomly masked during training (Zhang et al., 16 Feb 2026).

The second stage uses Conditional Flow Matching for waveform reconstruction. Let $$\langle \text{disease\_label}, \text{domain} \rangle$$8 be the ground-truth mel spectrogram and $$\langle \text{disease\_label}, \text{domain} \rangle$$9. The interpolation is

$B$0

A velocity field $B$1 is learned so that $B$2, where the condition $B$3 contains a content stream from temporally upsampled unit embeddings and a global timbre stream given by time-averaged BEATs. The flow-matching loss is

$B$4

A short prefix of the real mel is exposed in the condition to encourage continuity, and the final waveform is recovered with Vocos (Zhang et al., 16 Feb 2026).

The generator is designed to decouple pathological content from acoustic style. In the reported disentanglement evaluations, style-swap experiments that fix pathology and vary style reference achieve Style-Sim $B$5, Pathology-Acc $B$6, and FAD $B$7. Content-swap experiments that fix style and vary pathology achieve Style-Sim $B$8, Pathology-Acc $B$9, and FAD $^2$0. In a separate comparison at 32 steps, flow matching yields FAD 1.13 versus 1.31 for DDPM, style-sim 0.92 versus 0.90, and 0.6$^2$1 inference time. A style-token ablation over $^2$2 shows monotonic gains, with $^2$3 best (Zhang et al., 16 Feb 2026).

4. Resp-229k benchmark corpus

Resp-229k is a multimodal, cross-domain dataset intended to stress-test generalization under both data scarcity and long-tailed class imbalance. It aggregates 229,101 quality-controlled recordings, corresponding to approximately 408 hours, from five public sources. The construction described in the source summary is: UK COVID-19, ICBHI, and SPRSound for train/validation, with KAUH and COUGHVID reserved for test only. The taxonomy contains 16 classes, defined as 15 diseases plus 1 healthy control after consolidating severity-specific and synonymous labels (Zhang et al., 16 Feb 2026).

Split	Files	Hours
Train	196,654	341
Val	16,931	31
Test (cross-domain)	15,516	36

The class distribution is markedly imbalanced. From the reported raw count of $^2$4k before quality control, the Control Group has 156,527 recordings, COVID-19 has 77,994, Pneumonia has 1,909, and many long-tail classes have fewer than 200 examples. The paired text modality consists of concise clinical summaries synthesized from available metadata via DeepSeek-R1-Distill-Qwen-7B and audited by DeepSeek-V3.2-Exp plus human review (Zhang et al., 16 Feb 2026).

The benchmark supports both classification and generation tasks. For classification, two settings are stated: the 4-class ICBHI benchmark with classes Normal, Crackle, Wheeze, and Both, and the 16-class cross-domain classification task evaluated with Accuracy and macro-F1. For generation, the reported metrics are acoustic fidelity, using FAD and style cosine, and clinical event fidelity, using a held-out Diagnoser (Zhang et al., 16 Feb 2026).

5. Reported empirical performance

On the official ICBHI 4-class 60/40 split, Resp-Agent with LLM plus Longformer achieves specificity 79.29%, sensitivity 66.10%, and score 72.70%. The cited prior SOTA, MVST AST, has score 66.55%. On the Resp-229k cross-domain test in the original imbalanced regime, an audio-only Conformer yields Acc = 0.7200 and Macro-F1 = 0.1935, whereas the Resp-Agent Diagnoser yields Acc = 0.8494 and Macro-F1 = 0.2118, an improvement of +0.0183 F1 (Zhang et al., 16 Feb 2026).

Under the balanced regime with generative augmentation, the same comparison becomes more pronounced. The Conformer improves from 0.7200 / 0.1935 to 0.7820 / 0.5360, and Resp-Agent improves from 0.8494 / 0.2118 to 0.8870 / 0.5980. The planner comparison under a fixed $^2$5k synthetic clips on Test-CD gives a ranking of CE only at 0.212, Random at 0.442, Class-Prior at 0.512, Uncertainty static at 0.546, and Thinker-A$^2$6CA at 0.598 (Zhang et al., 16 Feb 2026).

Planner-factor ablations at $^2$7k further separate curriculum components: RareOnly gives 0.489, HardCaseOnly 0.512, Rare$^2$8HardDomain 0.528, and full Thinker-A$^2$9CA 0.541. Generator ablations also indicate that augmentation quality matters. Naïve audio augmentations degrade Conformer Macro-F1 from 0.1935 to 0.1688. Objective generation baselines, namely c-WaveGAN, AudioLDM 2, and StableAudio Open, produce style-sim $F1_i$0, FAD $F1_i$1, and weaker downstream gains with Conformer F1 $F1_i$2. Resp-Agent reports style-sim 0.92, FAD 1.13, and Conformer F1 = 0.5360 (Zhang et al., 16 Feb 2026).

These results support three narrower conclusions stated in the source summary: a closed-loop, LLM-orchestrated curriculum is more effective than static heuristics for augmentation under severe imbalance; modality weaving with anchor-based global attention outperforms unimodal and shallow-fusion alternatives; and a retooled LLM combined with conditional flow matching produces high-fidelity, style-controllable audio that improves diagnostic robustness on rare classes and across domains (Zhang et al., 16 Feb 2026).

6. Interpretation, scope, and terminological clarification

Resp-Agent uses the term “agent” in a specific architectural sense. The planner, generator, and diagnoser are coordinated components in a closed-loop optimization system rather than independent clinical actors. A common misunderstanding would be to reduce the framework to a conventional augmentation pipeline. The reported planner comparison does not support that reading: random, class-prior, and static-uncertainty synthesis all underperform Thinker-A$F1_i$3CA at the same synthesis budget, and naïve audio augmentations can reduce Macro-F1 (Zhang et al., 16 Feb 2026).

A second misunderstanding would be to treat Resp-Agent as an audio-only classifier with optional metadata. The diagnoser is explicitly multimodal at the input layer, with woven EHR summaries and audio embeddings processed jointly through Strategic Global Attention. The reported ablations show that raw metadata, late fusion, or weaving without anchors do not match the full weaving-plus-anchors-plus-LLM-EHR configuration (Zhang et al., 16 Feb 2026).

The name itself also requires disambiguation. “Resp-Agent” has appeared previously in an unrelated context as the Rule Responder eScience middleware, a rule-based, event-driven framework built around Reaction RuleML, Mule ESB, and the Prova engine (Paschke et al., 2010). In current respiratory-auscultation research, however, Resp-Agent denotes the multimodal generation-and-diagnosis system introduced in 2026 (Zhang et al., 16 Feb 2026). The overlap is nominal rather than methodological.

Within respiratory sound analysis, the main significance of Resp-Agent lies in how it couples representation design and data generation. The Modality-Weaving Diagnoser addresses the representation gap by preserving long-range clinical context and millisecond-level transients, while the Flow Matching Generator addresses the data gap by synthesizing hard-to-diagnose samples conditioned on pathology and acoustic style. A plausible implication is that the framework’s contribution is less a single model component than the joint operation of curriculum planning, multimodal token-level fusion, and controllable synthesis under cross-domain evaluation (Zhang et al., 16 Feb 2026).