CoSTA: Cognitive-State-Conditioned TTS Data Augmentation Using ASR Transcripts for Alzheimer's Disease Detection

Abstract: Speech-based Alzheimer's Disease (AD) detection is constrained by scarce pathological speech data. To address this, we propose CoSTA, a Text-to-Speech (TTS)-based data augmentation framework. Specifically, we first develop two Cognitive-State-Conditioned (CS-Cond) TTS models by adapting CosyVoice2 and F5-TTS to synthesize speech with distinct AD and Healthy Control characteristics. Furthermore, by constructing a transcript pool comprising Manual Transcripts (MT) and 36 Automatic Speech Recognition (ASR) transcripts, we investigate the impact of text sources on TTS-based augmentation. We also perform augmentation-factor analysis and test-time augmentation. Experiments on the ADReSS dataset show that CS-Cond TTS significantly improves synthetic speech utility, and ASR-driven augmentation frequently outperforms MT-driven augmentation. Finally, CoSTA yields a 4.16% gain over the baseline, achieving an audio-only accuracy of 85.83% on the ADReSS test set and outperforming prior methods.

• The paper presents a novel cognitive-state-conditioned TTS framework that leverages ASR transcripts for effective data augmentation in Alzheimer's detection.
• It integrates specialized TTS models, diverse transcript sources, and test-time augmentation to improve AD classification while addressing data scarcity.
• Empirical results demonstrate an audio-only accuracy of 85.83%, highlighting the value of preserving diagnostically significant ASR errors in synthetic speech.

CoSTA: Cognitive-State-Conditioned TTS Data Augmentation Using ASR Transcripts for Alzheimer's Disease Detection

Motivation and Context

Speech-based detection of Alzheimer's Disease (AD) is limited by the scarcity of pathological speech data due to privacy restrictions and limited patient access. Existing data augmentation (DA) techniques—primarily signal-level perturbations—fail to introduce diagnostically meaningful speech variations and sometimes degrade detection performance by generating indiscriminate distortions. Recent advances in TTS provide the means to synthesize diverse speech samples, yet standard TTS models are optimized for naturalness and intelligibility, inherently glossing over AD-specific speech characteristics. Furthermore, the value of text sources used for TTS augmentation, specifically ASR transcripts versus manual transcripts (MT), remains underexplored. The CoSTA framework introduces Cognitive-State-Conditioned (CS-Cond) TTS for targeted DA and systematically investigates the influence of transcript source, delivering substantially improved AD detection performance.

Figure 1: Overview of the CoSTA framework including CS-Cond TTS, transcript pool construction, DA strategies, and the audio-based AD classifier.

Methodology

CoSTA consists of four integrated components: CS-Cond TTS model development, transcript pool construction, DA strategies (including test-time augmentation), and a high-capacity audio-based AD detection model.

CS-Cond TTS Models

Two TTS models are adapted for cognitive-state conditioning: CosyVoice2 and F5-TTS.

• CosyVoice2 leverages LLM-based instruction fine-tuning to synthesize speech reflecting either AD or healthy control (HC) cognitive traits. Natural language instructions are prepended to transcripts to explicitly condition the speech generation process.
• F5-TTS applies a non-autoregressive, diffusion-based approach for speech synthesis, incorporating cognitive labels into the conditioning context via ConvNeXtv2 layers with RoPE encoding.

Both models undergo targeted training on respective AD and HC subsets. During inference, cognitive instructions or labels, original transcripts, and reference speech are supplied to generate class-consistent synthetic speech.

Transcript Pool Creation

A pool of 37 transcripts per sample is constructed, comprising MT and outputs from 36 ASR models (18 pretrained and 18 fine-tuned) representing diverse architectures (Wav2Vec2, HuBERT, WavLM, Whisper). This ensures broad coverage of transcription error distributions, which are hypothesized to encode diagnostically relevant cues.

Figure 2: Mean Word Error Rate (WER) for 36 ASR models, showing span and improvement via fine-tuning.

Data Augmentation Strategy

Synthetic speech samples are generated using various combinations of TTS models and transcript sources:

• Self-Reference Synthesis retains timbre by using the original speech as reference; suitable for $2\times$ DA.
• Intra-Class Cross-Synthesis further diversifies the data by combining transcripts from one subject with speech characteristics from another subject in the same class, facilitating augmentation factors beyond $2\times$.

Test-Time Augmentation (TTA) is also introduced, in which synthetic variants are generated at inference-time and ensembled with original predictions.

Speech-Based AD Detection Model

The classifier is based on a pretrained WavLM, employing hierarchical feature fusion and attentive pooling for robust representation extraction, followed by an MLP for binary AD/HC classification.

Experimental Setup

The ADReSS dataset is used for evaluation, containing 108 subjects for training (segregated into TTS and detection splits) and 48 subjects for testing, each providing speech-MT pairs. ASRs are fine-tuned on DementiaBank-derived subsets. Objective TTS quality metrics include Mel Cepstral Distortion (MCD), log-$F_0$ RMSE, and Frechet Audio Distance (FAD).

Results

CS-Cond TTS Efficacy

CS-Cond TTS models consistently achieve lower objective error metrics compared to their unconditioned pretrained counterparts across both AD and HC classes, indicating superior acoustic distribution matching.

Transcript Source Diversity

Fine-tuned ASRs demonstrate substantial WER improvement and provide a broad error spectrum (26.36%—68.55%), making ASR transcripts an effective source of linguistic diversity for DA.

AD Detection Performance

Augmentation using CS-Cond TTS and ASR transcripts yields marked accuracy improvements. Specifically, CS-Cond CosyVoice2 outperforms the baseline in 28/37 configurations, while CS-Cond F5-TTS does so in 24/37. ASR-driven augmentation is superior to MT-driven augmentation in the majority of cases across all TTS models.

Augmentation Factor Analysis

DA effectiveness follows an inverted-U profile: optimal performance is achieved at an augmentation factor of $2\times$, with excessive synthetic data leading to diminished performance owing to model overfitting to TTS artifacts.

Figure 3: AD detection accuracy as a function of augmentation factor, showing optimal results in the $2\times$ regime using CS-Cond CosyVoice2 and high-performing ASR transcripts.

Test-Time Augmentation

TTA enhances model accuracy by approximately 1%, attaining a peak audio-only accuracy of 85.83% on the ADReSS test set—a 4.16% absolute improvement over the baseline and surpassing previous audio-only methods.

Implications and Future Directions

The study demonstrates that cognitive-state conditioning in TTS, combined with the deliberate use of ASR error-rich transcripts, leads to significantly more effective DA for pathological speech modeling. The findings imply that non-random ASR errors act as valuable proxies for diagnostic speech traits, and their preservation in synthesized data is beneficial. Practically, CoSTA enables robust AD detection with minimal original data, addressing a core barrier in clinical speech-based diagnostics. Theoretically, the results challenge the prevailing assumption that DA should minimize transcript error, highlighting the utility of semantically meaningful transcription perturbations.

Future directions include cross-lingual synthesis for broader applicability, as well as leveraging synthetic pathological data to train multimodal and unified speech understanding models. The approach may generalize to other neurological conditions involving characteristic speech patterns and represents a substantive advance in DA methodology for medical speech applications.

Conclusion

CoSTA introduces a cognitively-conditioned TTS-based DA pipeline that systematically explores the utility of ASR-driven versus MT-driven transcript sources for robust AD detection. The framework delivers strong numerical results, with an audio-only accuracy of 85.83% on the ADReSS test set, achieved through CS-Cond TTS models and ASR transcript utilization, outperforming both traditional DA and prior state-of-the-art methods. The research has substantial methodological and practical implications for speech-based diagnostic modeling and lays the groundwork for future advancements in synthetic pathological speech generation and detection.