Resp-229k: Multimodal Respiratory Benchmark

• Resp-229k is a benchmark comprising 229k respiratory recordings paired with standardized clinical narratives across 16 diagnostic categories.
• It unifies heterogeneous public datasets under a multimodal, source-disjoint evaluation protocol to ensure robust cross-domain testing.
• The dataset supports tasks like disease classification, controllable audio generation, and synthetic rebalancing, mitigating long-tail imbalances.

Searching arXiv for recent and directly related papers on Resp-229k, Resp-Agent, and adjacent respiratory-audio benchmarks. I’ll look up the Resp-Agent paper and a few adjacent respiratory-audio papers on arXiv to ground the article with fresh citations. Resp-229k is a large-scale, multi-source, cross-domain benchmark for multimodal respiratory sound research, introduced as the data foundation of Resp-Agent and designed to address weak multimodal supervision, severe class imbalance, and unrealistic evaluation protocols in respiratory auscultation AI (Zhang et al., 16 Feb 2026). In the benchmark’s main description, it contains 229,101 effective samples totaling 408 hours, spans 16 diagnostic categories, pairs each recording with a standardized clinical narrative synthesized from available source fields, and enforces a strict source-disjoint protocol in which training and validation data come from sources different from those used for testing (Zhang et al., 16 Feb 2026). Its role is not limited to classification: the corpus is used for multimodal disease diagnosis, controllable respiratory sound generation, synthetic rebalancing, and cross-source robustness studies. A plausible implication is that Resp-229k is intended as both a dataset and an evaluation framework, rather than a passive aggregation of public recordings.

1. Definition, scope, and benchmark rationale

Resp-229k is presented as a benchmark corpus of “approximately 408 hours and 229k respiratory recordings spanning 16 diagnostic categories,” with the exact post-QC total given as 229,101 samples (Zhang et al., 16 Feb 2026). The benchmark is positioned explicitly against common limitations of prior respiratory auscultation datasets: small scale, single-source collection, poor balance, and limited aligned clinical context. Its construction therefore emphasizes three properties simultaneously: scale, multimodality, and cross-domain evaluation.

The benchmark is multimodal in a specific sense. Each respiratory audio clip is paired with a standardized clinical summary synthesized from existing metadata rather than with raw hospital EHR text. Section 3 clarifies that these summaries are created from “available source fields,” and Appendix E specifies that the process is data-to-text rather than audio-to-text (Zhang et al., 16 Feb 2026). This distinction matters because the textual modality is intended to normalize heterogeneous public metadata into a consistent paragraph style without inferring clinical facts from the waveform itself.

Resp-229k is also defined by its evaluation protocol. Training and validation are restricted to UK COVID-19, ICBHI, and SPRSound, whereas testing is performed exclusively on COUGHVID and KAUH, which are unseen during training (Zhang et al., 16 Feb 2026). The paper repeatedly describes this as source-disjoint, out-of-domain, and cross-institution/cross-device evaluation. This suggests that the benchmark is designed to measure robustness to changes in institution, sensor, and acquisition regime rather than only in-distribution pattern recognition.

The benchmark should not be conflated with deployment-oriented respiration-rate estimation systems. For example, EarResp-ANS targets window-level respiration rate estimation from in-ear microphones under realistic noise and reports rate-estimation metrics rather than 16-class disease diagnosis or multimodal text-audio supervision (Küttner et al., 3 Feb 2026). Resp-229k therefore occupies a different problem setting: clinically contextualized respiratory sound understanding under source shift.

2. Corpus composition, source allocation, and label space

Resp-229k is curated from five public datasets: UK COVID-19, ICBHI, SPRSound, COUGHVID, and KAUH (Zhang et al., 16 Feb 2026). The raw aggregated corpus contained 238,074 clips; after removing corrupted or too-short recordings, 229,101 clips remain for the main-paper experiments (Zhang et al., 16 Feb 2026). Table 1(a) reports the global split statistics.

Split	Files	Hours
Train	196,654	341
Validation	16,931	31
Test	15,516	36

The mean duration is 6.4 s overall, with 6.2 s for training, 6.6 s for validation, and 8.4 s for test (Zhang et al., 16 Feb 2026). Source allocation is central to the benchmark definition rather than an implementation detail. UK COVID-19, ICBHI, and SPRSound contribute only to training and validation, whereas COUGHVID and KAUH are test-only (Zhang et al., 16 Feb 2026). The paper specifies the recording conditions for these sources: UK COVID-19 uses microphone recordings at 48 kHz with mean duration 5.9 s; ICBHI uses stethoscope recordings at 4–44.1 kHz with mean duration 22.2 s; SPRSound uses stethoscope recordings at 8 kHz with mean duration 11.0 s; COUGHVID is microphone-based at 48 kHz with mean duration 6.9 s; and KAUH is stethoscope-based at 4 kHz with mean duration 15.0 s (Zhang et al., 16 Feb 2026).

The label space is unified into 16 classes: 15 disease categories plus 1 healthy control (Zhang et al., 16 Feb 2026). The raw label space inherited from the constituent datasets initially contained 20 classes, which were programmatically consolidated before splitting. Appendix A gives explicit examples of the consolidation rules: “Bronchiectasia” is mapped to “Bronchiectasis,” “Acute upper respiratory infection” is mapped to “URTI,” and severity-specific pneumonia labels are merged into “Pneumonia” (Zhang et al., 16 Feb 2026).

The final 16 classes are Airway foreign body, Asthma, Bronchiectasis, Bronchiolitis, Bronchitis, COPD, COVID-19, Chronic cough, Hemoptysis, Kawasaki disease, LRTI, Pneumonia, Pulmonary hemosiderosis, URTI, Other respiratory diseases, and Control Group (Zhang et al., 16 Feb 2026). The benchmark is explicitly long-tailed. In the raw 238,074-recording label space after unification but before QC, Control Group has 156,527 clips, COVID-19 has 77,994, and Pneumonia has 1,909 (Zhang et al., 16 Feb 2026). The authors describe this as “severe long-tail imbalance” and “extreme class imbalance,” with most diagnostic categories in the long tail.

3. Clinical narratives and multimodal quality control

The second modality in Resp-229k consists of standardized clinical narratives synthesized from structured metadata. The paper describes the benchmark as pairing each sample with “a clinical narrative synthesized from EHR records using LLMs and refined for accuracy,” while later clarifying that these narratives are generated from “available source fields” rather than from raw hospital EHR in the conventional sense (Zhang et al., 16 Feb 2026). Appendix E makes the provenance explicit: the process is data-to-text, not audio-to-text.

Two summary regimes are retained intentionally as part of the modeling challenge (Zhang et al., 16 Feb 2026). One regime is technical or event-driven, containing fields such as auscultation events, recording site, sensor or filter information, breathing phases, wheezes or crackles, and acquisition context. The other is clinically enriched, containing demographics, smoking status, comorbidities, symptoms, and past medical history when available. The summaries therefore “adapt to source coverage”: when richer metadata exist, they are included; when only acquisition context or auscultation-event fields exist, the summary focuses on those (Zhang et al., 16 Feb 2026).

The text-generation engine used for dataset construction is DeepSeek-R1-Distill-Qwen-7B, described as a “lightweight data-to-text engine” that consolidates heterogeneous CSV, TXT, JSON, and filename-derived metadata into a schema-grounded paragraph with a consistent style across sources (Zhang et al., 16 Feb 2026). The model “does not interpret audio,” and the validator prompt strictly forbids inventing patient metadata such as age, sex, or comorbidities and forbids altering high-level pathology labels (Zhang et al., 16 Feb 2026).

Quality control over the narratives is unusually detailed. All 238,074 generated summaries undergo a two-stage audit (Zhang et al., 16 Feb 2026). Stage 1 performs heuristic pre-screening and flags EMPTY_OR_TRUNCATED, OVERLONG, and PROMPT_LEAK. Stage 2 uses DeepSeek-V3.2-Exp to audit all suspicious records from Stage 1 plus a 1% random sample of heuristically clean records (Zhang et al., 16 Feb 2026). The reported audit statistics are:

Audit item	Count
Total records	238,074
Heuristic suspicious	3,356
Randomly audited clean samples	2,300
Total sent to QA LLM	5,656
LLM kept as OK	3,766
LLM rewrote	1,774

The effective rewrite rate is 0.7451%, computed as 1,774 / 238,074 (Zhang et al., 16 Feb 2026). Of the 1,774 rewrites, 1,747 were due to OVERLONG_OR_PROMPT_LEAK rather than substantive hallucination, and all 1,774 LLM-proposed rewrites underwent manual review by the authors (Zhang et al., 16 Feb 2026). This suggests that the benchmark’s textual side is engineered for consistency and provenance control rather than unconstrained synthetic augmentation.

4. Formal tasks, protocol design, and data standardization

Resp-229k formally specifies two benchmark tasks: multimodal disease classification, evaluated with accuracy and macro-F1; and controllable audio generation conditioned on disease semantics, evaluated with objective acoustic similarity and clinical-event fidelity (Zhang et al., 16 Feb 2026). In practice, the experiments broaden this into a larger suite that includes long-tailed recognition, robustness under varying synthetic budgets, source-disjoint Test-CD evaluation, leave-one-source-out analysis across all five constituent datasets, and style-swap/content-swap tests for generation (Zhang et al., 16 Feb 2026).

For diagnosis experiments, the waveform input $x$ is standardized to 10-second, 16 kHz clips via crop/pad, pretrained BEATs features are extracted, and the audio block length is fixed at $T=496$ frames (Zhang et al., 16 Feb 2026). The paper gives the woven audio-token construction as

$$E_{[A]} \leftarrow W \,\mathrm{Align}\!\left(\Phi_{\mathrm{BEATS}}(x)\right), \quad W \in \mathbb{R}^{D \times H}, \; E_{[A]} \in \mathbb{R}^{T \times H}.$$

Here, $\mathrm{Align}(\cdot)$ deterministically crops or pads the BEATs features to $T=496$ time steps, and $W$ projects them into the Longformer hidden space (Zhang et al., 16 Feb 2026).

The temporal motivation is explicit. A 10 s clip over $T=496$ steps yields a per-step hop of approximately 20.1613 ms, and with anchor stride $s=4$, adjacent global audio anchors are spaced at approximately 80.6452 ms, with worst-case snap error to the nearest anchor bounded by approximately 40.3 ms (Zhang et al., 16 Feb 2026). The strategic global attention pattern is defined by

$$A := \{a_{1+ks} \mid k = 0,1,\ldots,\lfloor (T-1)/s \rfloor\},$$

$$G := \{\mathrm{pos}([\mathrm{CLS}]), \mathrm{pos}([\mathrm{DESCRIPTION}])\} \cup A.$$

This attention design is used to preserve both long-range text context and millisecond-scale acoustic transients (Zhang et al., 16 Feb 2026).

For the audio-only Conformer baseline, the paper instead uses 10-second, 16 kHz clips converted to 128-bin log-mel spectrograms with a 1024-point FFT, window length 1024 samples, hop length 160 samples, frequency range 50–8000 Hz, and mean/std normalization (Zhang et al., 16 Feb 2026). For the text-only baseline, Appendix C reports tokenization with minimum word frequency 2, truncation or padding to 100 tokens, and a BiLSTM encoder (Zhang et al., 16 Feb 2026). These preprocessing choices matter because Resp-229k is used to compare multimodal, audio-only, and text-only formulations under the same source-disjoint benchmark.

5. Dataset-enabled modeling: diagnosis, generation, and synthetic balancing

Resp-229k is the training and evaluation substrate for both the diagnoser and the generator in Resp-Agent (Zhang et al., 16 Feb 2026). For the generator, the paired text-audio examples are used to combine disease semantics from text with acoustic style from a 10 s, 16 kHz reference audio clip. The style-conditioning mechanism is defined as

$T=496$0

where $T=496$1 are BEATs frame features from the reference audio, $T=496$2 compresses them into $T=496$3 style descriptors, and $T=496$4 maps them into the LLM hidden space (Zhang et al., 16 Feb 2026).

The discrete unit-generation objective is given as

$T=496$5

with $T=496$6 denoting acoustic units, $T=496$7 the disease-semantics text, and $T=496$8 the style embeddings (Zhang et al., 16 Feb 2026). The conditional flow-matching reconstruction stage is defined by

$T=496$9

and

$$E_{[A]} \leftarrow W \,\mathrm{Align}\!\left(\Phi_{\mathrm{BEATS}}(x)\right), \quad W \in \mathbb{R}^{D \times H}, \; E_{[A]} \in \mathbb{R}^{T \times H}.$$0

These formulations matter because Resp-229k is not merely an evaluation set; it provides the aligned supervision required for controllable respiratory sound synthesis (Zhang et al., 16 Feb 2026).

The benchmark is also used for active synthetic rebalancing. Table 3 reports planner policies under a synthetic budget $$E_{[A]} \leftarrow W \,\mathrm{Align}\!\left(\Phi_{\mathrm{BEATS}}(x)\right), \quad W \in \mathbb{R}^{D \times H}, \; E_{[A]} \in \mathbb{R}^{T \times H}.$$1 allocated over label and domain combinations on Resp-229k/Test-CD (Zhang et al., 16 Feb 2026). Under matched $$E_{[A]} \leftarrow W \,\mathrm{Align}\!\left(\Phi_{\mathrm{BEATS}}(x)\right), \quad W \in \mathbb{R}^{D \times H}, \; E_{[A]} \in \mathbb{R}^{T \times H}.$$2, No-Synth (CE) yields Acc 0.849, Macro-F1 0.212, and Macro-F1_tail 0.074, whereas Thinker-A$$E_{[A]} \leftarrow W \,\mathrm{Align}\!\left(\Phi_{\mathrm{BEATS}}(x)\right), \quad W \in \mathbb{R}^{D \times H}, \; E_{[A]} \in \mathbb{R}^{T \times H}.$$3CA yields 0.887, 0.598, and 0.421, respectively (Zhang et al., 16 Feb 2026). On leave-one-source-out evaluation, the paper reports average Macro-F1 / tail values of 0.237 / 0.086 for No-Synth, 0.473 / 0.334 for Class-Prior, and 0.532 / 0.383 for Thinker-A$$E_{[A]} \leftarrow W \,\mathrm{Align}\!\left(\Phi_{\mathrm{BEATS}}(x)\right), \quad W \in \mathbb{R}^{D \times H}, \; E_{[A]} \in \mathbb{R}^{T \times H}.$$4CA (Zhang et al., 16 Feb 2026). These numbers indicate that Resp-229k is explicitly used to study long-tail mitigation under source shift, not only absolute classification accuracy.

6. Empirical findings, interpretation, and limitations

The benchmark’s reported results show a clear hierarchy among modalities on the original imbalanced Test-CD split. Text-only models perform poorly, audio-only models are substantially stronger, and multimodal modeling gives additional gains (Zhang et al., 16 Feb 2026). Table 6 reports 0.0912 accuracy / 0.0401 macro-F1 for an LSTM text-only model, 0.1585 / 0.0813 for Longformer text-only, 0.7200 / 0.1935 for the Conformer audio-only baseline, and 0.8494 / 0.2118 for the Resp-Agent diagnoser (Zhang et al., 16 Feb 2026). The paper interprets this as evidence that the narratives alone are insufficient and that the main value lies in cross-modal grounding.

The strongest dataset-driven result concerns balancing. Table 8 compares original and balanced regimes on Resp-229k (Zhang et al., 16 Feb 2026). For the full multimodal model, performance improves from 0.8494 / 0.2118 to 0.8870 / 0.5980 in accuracy / macro-F1; for the audio-only Conformer, it improves from 0.7200 / 0.1935 to 0.7820 / 0.5360; and for the text-only LSTM, from 0.0912 / 0.0401 to 0.3020 / 0.2140 (Zhang et al., 16 Feb 2026). The macro-F1 jump from 0.2118 to 0.5980 for the full model is one of the benchmark’s central findings because it quantifies the effect of severe long-tail imbalance.

The textual side of the benchmark is also empirically consequential. In the diagnoser ablation on Test-CD, late fusion with raw metadata achieves 0.780 / 0.145, whereas late fusion with LLM EHR summaries reaches 0.790 / 0.160; modality weaving with raw metadata and anchors gives 0.835 / 0.195, and the full diagnoser with LLM EHR plus anchors reaches 0.849 / 0.212 (Zhang et al., 16 Feb 2026). This indicates that the LLM-distilled narratives are not merely decorative. Their value, however, depends on the fusion mechanism and attention pattern.

Generation experiments likewise use Resp-229k as a benchmark for clinically meaningful synthesis. In Table 10, Resp-Agent reports cosine similarity $$E_{[A]} \leftarrow W \,\mathrm{Align}\!\left(\Phi_{\mathrm{BEATS}}(x)\right), \quad W \in \mathbb{R}^{D \times H}, \; E_{[A]} \in \mathbb{R}^{T \times H}.$$5 and FAD 1.13, compared with $$E_{[A]} \leftarrow W \,\mathrm{Align}\!\left(\Phi_{\mathrm{BEATS}}(x)\right), \quad W \in \mathbb{R}^{D \times H}, \; E_{[A]} \in \mathbb{R}^{T \times H}.$$6 and 1.54 for StableAudio Open fine-tuned, $$E_{[A]} \leftarrow W \,\mathrm{Align}\!\left(\Phi_{\mathrm{BEATS}}(x)\right), \quad W \in \mathbb{R}^{D \times H}, \; E_{[A]} \in \mathbb{R}^{T \times H}.$$7 and 1.92 for AudioLDM 2 fine-tuned, and $$E_{[A]} \leftarrow W \,\mathrm{Align}\!\left(\Phi_{\mathrm{BEATS}}(x)\right), \quad W \in \mathbb{R}^{D \times H}, \; E_{[A]} \in \mathbb{R}^{T \times H}.$$8 and 2.85 for c-WaveGAN (Zhang et al., 16 Feb 2026). When training downstream classifiers on balanced synthetic data, the Resp-Agent-balanced regime also gives the strongest downstream performance on Resp-229k/Test-CD for both multimodal Longformer and audio-only Conformer models (Zhang et al., 16 Feb 2026).

Several limitations remain explicit or implied in the benchmark design. Metadata are heterogeneous across sources, and the text modality is uneven because some summaries include demographics and symptoms while others contain only technical recording context or auscultation-event fields (Zhang et al., 16 Feb 2026). Subject-level counts, patient-overlap analysis, and hospital-by-hospital demographic summaries are not reported in the provided text (Zhang et al., 16 Feb 2026). The long tail is scientifically deliberate but makes raw performance highly sensitive to majority classes unless macro-averaged metrics or balanced training are used. A plausible implication is that Resp-229k is best understood as a stress test for robust multimodal respiratory learning under source heterogeneity, rather than as a uniformly annotated clinical cohort.

From an ethical and release perspective, the paper states that all source data are public and previously de-identified, no new human-subject data were collected, and no PII or PHI were accessed or processed (Zhang et al., 16 Feb 2026). It further provides release endpoints for code, the curated dataset, and trained checkpoints. In that sense, Resp-229k is positioned as an open research benchmark whose significance derives from the combination of scale, audited multimodal alignment, unified taxonomy, and source-disjoint evaluation (Zhang et al., 16 Feb 2026).