MoTAS: Multimodal Early AD Screening

• MoTAS is a multimodal framework for early Alzheimer’s screening that uses class-consistent TTS augmentation and Mixture-of-Experts feature selection to refine acoustic and textual cues.
• It processes spontaneous speech by automatic transcription, extracting MFCC, spectrogram, Wav2Vec2, and BERT embeddings, then fusing them through an MLP classifier to achieve 85.71% accuracy on ADReSSo.
• The framework demonstrates a synergistic effect where 2× TTS augmentation combined with adaptive MoE refinement optimizes AD recall and robustness even under limited-data conditions.

MoTAS is a multimodal framework for early screening of Alzheimer’s Disease (AD) from spontaneous speech that combines Text-to-Speech (TTS) augmentation with Mixture-of-Experts (MoE)-guided feature selection to improve robustness under limited-data conditions. In the formulation reported for the ADReSSo benchmark, the pipeline begins with automatic speech recognition (ASR), augments the speech set through class-consistent TTS resynthesis, extracts acoustic and textual embeddings, and uses an MoE mechanism to refine modality-specific representations before multimodal fusion and binary classification into AD versus cognitively normal (CN) classes. On ADReSSo, MoTAS achieves an average accuracy of 85.71% over five runs and is reported as outperforming the listed baselines (Shao et al., 28 Aug 2025).

1. Definition and problem setting

MoTAS, expanded as MoE-Guided Feature Selection from TTS-Augmented Speech for Enhanced Multimodal Alzheimer’s Early Screening, targets non-invasive early AD screening from speech (Shao et al., 28 Aug 2025). The motivating problem is twofold: limited data availability in clinical speech corpora and the lack of fine-grained, adaptive feature selection in multimodal models. The framework addresses these constraints jointly by enlarging the effective training pool through TTS augmentation and by using MoE gating to select the most informative features across modalities.

The reported use case is based on raw “Cookie Theft” recordings. These recordings are segmented into sentence-level utterances, after which Whisper ASR converts each segment $s$ into a transcript $t = f_{\mathrm{ASR}}(s)$. The resulting system is explicitly multimodal: it operates on acoustic inputs and ASR-derived transcripts, and it combines hand-crafted or classical acoustic representations with pretrained neural embeddings.

A central design choice is that MoTAS does not treat all modalities symmetrically at the feature-selection stage. MFCC, spectrogram, and text embeddings are refined by MoE, whereas the Wav2Vec2 embedding is included in the final fusion without further selection. This suggests an architecture in which adaptive gating is applied where modality redundancy or noise is expected to be most consequential, while preserving a pretrained speech representation as a direct input to the classifier.

2. End-to-end processing pipeline

The high-level pipeline consists of five stages: data collection and ASR transcription, TTS-based data augmentation, feature extraction, MoE-guided feature selection, and feature fusion with classification (Shao et al., 28 Aug 2025).

First, the raw recordings are segmented into sentence-level utterances. Whisper ASR is then used to produce transcripts for each segment. This ASR stage is not merely ancillary: the transcripts are used both as textual inputs and as the basis for subsequent augmentation procedures.

Second, TTS augmentation is applied with a pretrained FishSpeech TTS model. Each speaker’s voice is re-synthesized with different transcripts from the same class to produce synthetic samples $\hat{s}$. The augmented pool is defined as

$$S_{\mathrm{aug}} = S_{\mathrm{orig}} \cup \{\hat{s}\},$$

and the augmented audio set is re-transcribed by ASR into $T_{\mathrm{aug}}$. The resynthesis procedure is specified more concretely by considering AD and CN subsets separately. Let $S_{\mathrm{AD}}$ and $S_{\mathrm{CN}}$ denote the original audio sets and $T_{\mathrm{AD}}$, $T_{\mathrm{CN}}$ their ASR transcripts. For each pair $(s_i, t_j)$ with $t = f_{\mathrm{ASR}}(s)$0, synthetic audio is generated as

$t = f_{\mathrm{ASR}}(s)$1

so that $t = f_{\mathrm{ASR}}(s)$2 preserves speaker $t = f_{\mathrm{ASR}}(s)$3’s voice but uses transcript $t = f_{\mathrm{ASR}}(s)$4.

Third, acoustic and text features are extracted. The feature inventory comprises MFCC embeddings via BiLSTM, spectrogram embeddings via ResNet-18, Wav2Vec2 embeddings, and BERT [CLS] embeddings derived from the ASR transcripts.

Fourth, MoE-guided feature selection operates on the MFCC, spectrogram, and text embeddings. The mechanism learns a gating function that weights multiple sub-networks, producing refined vectors for each of these modalities.

Fifth, the refined MoE outputs are concatenated with the Wav2Vec2 embedding and passed to a three-layer multilayer perceptron (MLP) with ReLU and dropout for binary prediction.

3. Modal representations and augmentation regime

The embedding definitions are given explicitly for each segment $t = f_{\mathrm{ASR}}(s)$5:

Representation	Extraction path	Dimension
MFCC embedding $t = f_{\mathrm{ASR}}(s)$6	$t = f_{\mathrm{ASR}}(s)$7	128
Spectrogram embedding $t = f_{\mathrm{ASR}}(s)$8	$t = f_{\mathrm{ASR}}(s)$9	1000
Wav2Vec2 embedding $\hat{s}$0	$\hat{s}$1	768
Text embedding $\hat{s}$2	$\hat{s}$3	1024

These representations define the multimodal input space of MoTAS (Shao et al., 28 Aug 2025). MFCC and spectrogram features capture distinct acoustic views, Wav2Vec2 contributes a pretrained speech embedding, and BERT encodes transcript-level linguistic content.

The augmentation process is controlled by an augmentation ratio $\hat{s}$4 that balances real and synthetic samples. The reported empirical finding is that $\hat{s}$5, described as doubling the set, gave the best validation accuracy. No bespoke loss term is introduced for TTS. Instead, quality control is attributed to re-ASR of the synthesized speech and to careful ratio selection.

The data-volume effect is substantial at the training-set level. The ADReSSo dataset is reported as having 166 training samples and 71 test samples, and after 3× TTS augmentation the training set increases to 481 samples. At the same time, the ablation results indicate that excessive augmentation can harm generalization. This is expressed directly in the study’s summary: augmentation beyond 2× reduces test accuracy relative to the best configuration.

A plausible implication is that MoTAS uses augmentation not as a generic regularizer but as a constrained class-conditional perturbation mechanism whose benefit depends on maintaining a favorable real-to-synthetic balance.

4. Mixture-of-Experts feature selection and classifier design

For each modality $\hat{s}$6, MoTAS deploys $\hat{s}$7 experts $\hat{s}$8 (Shao et al., 28 Aug 2025). The gating function is

$\hat{s}$9

where $$S_{\mathrm{aug}} = S_{\mathrm{orig}} \cup \{\hat{s}\},$$0 and $$S_{\mathrm{aug}} = S_{\mathrm{orig}} \cup \{\hat{s}\},$$1. If the expert outputs are denoted $$S_{\mathrm{aug}} = S_{\mathrm{orig}} \cup \{\hat{s}\},$$2, the fused MoE output for a modality is

$$S_{\mathrm{aug}} = S_{\mathrm{orig}} \cup \{\hat{s}\},$$3

The modality-specific refined vectors are given as

$$S_{\mathrm{aug}} = S_{\mathrm{orig}} \cup \{\hat{s}\},$$4

$$S_{\mathrm{aug}} = S_{\mathrm{orig}} \cup \{\hat{s}\},$$5

$$S_{\mathrm{aug}} = S_{\mathrm{orig}} \cup \{\hat{s}\},$$6

Regularization is applied through an entropy penalty on the gate output:

$$S_{\mathrm{aug}} = S_{\mathrm{orig}} \cup \{\hat{s}\},$$7

with $$S_{\mathrm{aug}} = S_{\mathrm{orig}} \cup \{\hat{s}\},$$8. The stated purpose is to encourage sparsity and push the gate to concentrate on a subset of experts.

After MoE refinement, the final feature vector is

$$S_{\mathrm{aug}} = S_{\mathrm{orig}} \cup \{\hat{s}\},$$9

This vector is fed into an MLP classifier with the layer sequence

$T_{\mathrm{aug}}$0

Training uses binary cross-entropy,

$T_{\mathrm{aug}}$1

and the total loss is

$T_{\mathrm{aug}}$2

The architecture therefore couples adaptive modality-specific refinement with a relatively standard discriminative head. This suggests that the main novelty is concentrated in the interaction between augmentation and MoE-based feature selection rather than in the terminal classifier.

5. Empirical performance on ADReSSo

The reported evaluation uses the ADReSSo dataset with 166 training samples and 71 test samples, and the metrics are Accuracy, Precision, Recall, and F1-score for each class (Shao et al., 28 Aug 2025). The paper lists the following baselines: eGeMAPS+SVM (64.79%), Wav2Vec2+TB (74.65%), Whisper-TL (77.46%), Late Fusion (78.87%), WavBERT variants, TDNN-ASR-M5 (84.51%), and Whisper-TL-FTP (84.51%).

MoTAS, averaged over five runs, attains:

Metric	Value
Accuracy	85.71%
AD Precision	80.49%
AD Recall	94.29%
AD F1	86.84%
CN Precision	93.10%
CN Recall	77.14%
CN F1	84.38%

No formal $T_{\mathrm{aug}}$3-values are reported, but the standard deviations over five seeds are stated to be under 1%, which is presented as evidence of stable gains.

The class-wise breakdown is clinically consequential within the authors’ framing. In particular, the AD recall of 94.29% is highlighted as minimizing false negatives, which is described as a priority in early screening to ensure timely follow-up diagnostics. This does not establish clinical deployment readiness by itself, but it does define the performance emphasis of the method: recall-sensitive screening rather than balanced downstream diagnosis.

6. Ablation structure, observed gains, and practical interpretation

The ablation study isolates the contributions of TTS and MoE on ADReSSo (Shao et al., 28 Aug 2025):

Configuration	Accuracy
No TTS, no MoE	78.28%
No TTS, MoE	79.71%
2× TTS, no MoE	81.72%
2× TTS, MoE	85.71%
1.5× TTS, MoE	81.72%
2.5× TTS, MoE	82.86%
3× TTS, MoE	80.29%

The accompanying interpretation in the source is explicit. MoE alone yields a modest gain of approximately +1.4%; 2× TTS alone yields approximately +3.4%; and the combined configuration yields approximately +7.4% relative to the non-augmented, non-MoE baseline. The study characterizes this as strong synergy between augmentation and adaptive feature selection. It also states that excessive augmentation, defined here as greater than 2×, hurts generalization.

The discussion section provides the main practical reading of these results. TTS augmentation is said to improve model robustness without collecting new clinical data. The MoE mechanism is described as learning to focus on the most discriminative acoustic and linguistic cues, with examples given as rhythm, pauses, and lexical deficits. The gated mechanism is further described as down-weighting unreliable ASR transcripts in impaired speech, thereby improving robustness to transcription errors common in real-world clinics.

Scalability is also addressed directly. The combination of Whisper ASR, FishSpeech TTS, pretrained encoders, and a lightweight MoE+MLP stack is described as runnable on commodity GPUs, with the stated implication of facilitating eventual deployment in screening tools. This suggests a design objective centered on practical screening workflows rather than solely benchmark optimization.

7. Scope, distinctions, and limitations of the reported evidence

MoTAS is a speech-based AD screening framework and should be distinguished from unrelated uses of similar acronyms in other domains. For example, “MOTS” denotes “Multi-Object Tracking and Segmentation,” a computer vision task centered on dense mask-based object tracking rather than clinical speech analysis (Voigtlaender et al., 2019). The similarity is lexical rather than methodological.

Within the evidence reported for MoTAS itself, several boundaries are explicit (Shao et al., 28 Aug 2025). The evaluation is on ADReSSo, and the central claim of performance superiority is relative to the baselines listed there. No formal $T_{\mathrm{aug}}$4-values are reported. The augmentation procedure is beneficial only up to a point, with the best validation accuracy obtained at $T_{\mathrm{aug}}$5 and weaker results at 2.5× and 3×. This constrains any interpretation that more synthetic data is uniformly advantageous.

The framework’s robustness claims are tied to specific design choices: re-ASR after synthesis, multimodal embeddings, and entropy-regularized gating across MFCC, spectrogram, and text features. The paper’s discussion presents these components as mechanisms for dealing with data scarcity, noisy transcripts, and redundant multimodal information. A plausible implication is that MoTAS is best understood not as a generic augmentation pipeline but as a coordinated strategy for data-limited multimodal screening, where augmentation and selective fusion are mutually dependent rather than independent modules.

In sum, MoTAS integrates TTS augmentation and MoE-guided multimodal fusion into a coherent pipeline for early AD screening from spontaneous speech. Its defining characteristics are class-consistent voice-preserving resynthesis, modality-specific expert gating, direct inclusion of Wav2Vec2 in the fused representation, and an empirical operating point that prioritizes AD recall while remaining sensitive to over-augmentation.