Structured Speaker-Deficiency Adaptation (SSDA)

• SSDA is an adaptive methodology for ASR that separates speaker identity and deficiency severity to robustly handle dysarthric and elderly speech.
• It employs two-layer bottleneck residual adapter blocks, enabling composable, data-efficient adaptation for both supervised fine-tuning and unsupervised test-time adjustments.
• The approach significantly reduces word error rates on benchmarks like UASpeech and DementiaBank, outperforming traditional and single-attribute adaptation methods.

Structured Speaker-Deficiency Adaptation (SSDA) is an adaptive methodology for Automatic Speech Recognition (ASR) designed to address the unique challenges presented by dysarthric and elderly speech. It leverages separate, composable model adapters for speaker identity and speech deficiency severity to produce foundation models that generalize robustly to unseen speakers and impairment profiles. SSDA is deployed in two main phases: a supervised adaptive fine-tuning stage that prepares a speaker- and deficiency-invariant base, and an unsupervised test-time adaptation procedure for rapid and data-efficient customization to new individuals (Hu et al., 2024).

1. Problem Motivation and Challenges

Dysarthric and elderly speech markedly diverge from normative adult speech due to phenomena such as articulatory imprecision, dysfluencies, neuro-motor impairments, slurred pronunciation, and cognitive pauses. These factors are compounded by the scarcity and pronounced heterogeneity of available data—typically only a few hours per speaker—spanning variable impairment severities, age, and gender. Conventional fine-tuning of SSL-based speech foundation models (SFMs) like HuBERT and wav2vec2-conformer on such data results in overfitting to the limited speakers present during training, thereby degrading generalization to unseen speakers. Existing speaker-adaptation methods (including LHUC, fMLLR, and Bayes LHUC) treat each speaker monolithically, neglecting the structured interplay between speaker identity and impairment level. SSDA explicitly models these distinct sources of variability using separate adapters, yielding highly generalizable ASR systems for sparse, diverse populations.

2. Model Architecture and Adapter Construction

SSDA operates on a backbone SFM—either HuBERT-large (960h) or wav2vec2-conformer—comprising a multi-layer CNN encoder, L transformer blocks, and a CTC output head. Adapter insertion is flexible, occurring after the CNN encoder (position 0) or within transformer block ℓ $(1 \leq ℓ \leq L)$.

Residual Adapter Block (RAB)

For both speaker and deficiency adaptation, SSDA employs a two-layer bottleneck residual adapter: $$f(h^ℓ;Θ_{ℓ}) = \mathrm{LN}\Bigl(\mathrm{Dropout}\bigl(P^u_{ℓ}\,\zeta(P^d_{ℓ} h^ℓ)\bigr)\Bigr)$$ where $P^d_{ℓ}$ and $P^u_{ℓ}$ are down- and up-projection matrices ($k \ll m$ for bottleneck dimension), $\zeta$ denotes GeLU activation, with optional dropout and layer normalization. The adapted output is $h^{ℓ,\mathrm{out}} = h^ℓ + f(h^ℓ;Θ_{ℓ})$.

Structured Combination

Each layer maintains two adapter parameter sets: $\Theta_{ℓ,d}$ for deficiency level $d$ and $\Theta_{ℓ,s}$ for speaker $s$. Deficiency adapters are applied first: $h^{ℓ,d} = h^ℓ + f(h^ℓ;Θ_{ℓ,d})$ followed by speaker adapters: $h^{ℓ,d,s} = h^{ℓ,d} + f(h^{ℓ,d};Θ_{ℓ,s})$ This design supports composability and a form of near-linear separation, as for most practical cases $h' \approx h + A_d(h) + A_s(h)$.

3. Supervised Adaptive Fine-Tuning (AFT)

In the AFT phase, SSDA learns a neutral base model invariant to speaker and deficiency via a two-stage adapter optimization per minibatch (using ground-truth transcripts):

1. Optimize deficiency adapters $\Theta_d$ for each deficiency group across its data, holding speaker adapters fixed.
2. Optimize speaker adapters $\Theta_s$ for each speaker’s data, holding deficiency adapters fixed.
3. Optionally alternate or repeat these stages.
4. Simultaneously fine-tune backbone parameters $\Phi$ over all multi-speaker/minority group data.

The training objective combines CTC loss and weight decay regularization: $$\mathcal{L}_{\text{AFT}}(\Phi, \Theta_d, \Theta_s) = \mathcal{L}_{CTC}(\Phi, \Theta_d, \Theta_s; \mathcal{D}) + \lambda_d \|\Theta_d\|^2 + \lambda_s \|\Theta_s\|^2$$ with $\lambda_d$, $\lambda_s$ preventing overfitting.

4. Unsupervised Test-Time Adaptation

For new speakers (lacking transcripts or explicit deficiency labels), SSDA adapts in two stages:

1. Deficiency Label Prediction: Apply a spectro-temporal classifier to the input audio for deficiency bin assignment $\hat{d}$.
2. Adapter Tuning:
   • Decode the test speaker’s audio using the neutral AFT model, generating a “hypothesis” transcript $\hat{Y}$.
   • Holding backbone $\Phi$ fixed, estimate $\Theta_{\hat{d}}$ by minimizing CTC loss over $(\mathrm{audio}, \hat{Y})$.
   • Fix $\Phi$, $\Theta_{\hat{d}}$; optimize speaker adapter $\Theta_{\hat{s}}$ similarly.

Only the adapter matrices (few million parameters) are updated during test; this means adaptation is fast and does not require supervised transcripts.

5. Experimental Setup and Baseline Comparisons

Experiments utilize two principal corpora:

• UASpeech dysarthric: 16 dysarthric and 13 control speakers; ~130h training and ~9h test; evaluation on Block 2 test set of dysarthric speakers.
• DementiaBank Pitt elderly: 292 elderly speakers; 58.9h training (augmented), 2.5h development, 0.6h evaluation; splits are speaker-disjoint.

Performance is measured via Word Error Rate (WER) $=\frac{S+D+I}{N_{ref}}$. SSDA is evaluated against:

• Standard SFM fine-tuning (no adapters).
• A global adapter shared by all test speakers.
• Single-attribute adapter (speaker-only or deficiency-only). Adapter bottleneck dimensions are $k=256$ (UASpeech); $k=128$ for speaker-adapters on DementiaBank, tuned for the lower per-speaker data regime.

6. Quantitative Results

The core results are summarized below:

System	UASpeech WER (%)	Δ Abs	Δ Rel
Fine-tuned HuBERT (no adapters)	27.71	–	–
Global adapter	26.73	-0.98	-3.54%
Speaker-only RAB	25.12	-2.59	-9.35%
Deficiency-only RAB	25.51	-2.20	-7.94%
Structured SD-RAB (SSDA)	24.70	-3.01	-10.86%

• On very low intelligibility speakers: WER $59.47 \rightarrow 57.38$ (–2.09 abs).
• On unseen words: WER $50.06 \rightarrow 44.60$ (–5.46 abs).
• Cross-system rescoring baseline (HuBERT+wav2vec2-conformer): SSDA reaches $19.45\%$ WER.

System	DementiaBank WER (%)	Δ Abs	Δ Rel
Fine-tuned Conformer	21.61	–	–
Speaker-only RAB	25.12†	+3.51	+16.2%†
Deficiency-only RAB	25.51†	+3.90	+18.0%†
Structured SD-RAB (SSDA)	20.11	-1.50	-6.94%

†Single-attribute adapters do not help; only the structured combination yields improvement.

7. Insights, Limitations, and Future Directions

By stratifying adaptation across speaker identity and deficiency severity, SSDA successfully captures orthogonal variability components that existing methods conflate. The supervised AFT stage yields an unbiased starting point for unsupervised adaptation, allowing effective quick customization per speaker at test time with minimal adaptation overhead. SSDA surpasses “no adapter,” “global adapter,” and “single-attribute” baselines on both dysarthric and elderly speech tasks (with state-of-the-art absolute WER drops up to 3.01% and 1.50% for UASpeech and DementiaBank, respectively). A plausible implication is that this adapter factorization could generalize to other low-resource, high-heterogeneity speech domains. Future research may explore online per-utterance adaptation and extension to new populations exhibiting diverse speech impairments (Hu et al., 2024).