Group-Aware Partial Model Merging (GRAPAM)

• The paper introduces GRAPAM, a strategy that clusters acoustic embeddings and performs group-specific partial fine-tuning to enhance pediatric ASR accuracy.
• It selectively updates submodules like FFN or ATTN, reducing updated parameters while achieving up to a 33.7% relative improvement in WER.
• Parameter-level merging integrates global and group-adapted models, effectively leveraging diverse children’s speech characteristics for better recognition.

Group-Aware Partial Model Merging (GRAPAM) is a parameter-efficient strategy designed to adapt large pre-trained Automatic Speech Recognition (ASR) models to children’s speech, addressing the pronounced acoustic variability and scarcity of data typical of pediatric ASR tasks. By combining unsupervised clustering of acoustic embeddings, group-specific partial fine-tuning, and parameter-level model interpolation, GRAPAM yields consistent improvements in recognition accuracy, outperforming conventional full-model fine-tuning while updating significantly fewer parameters (Rolland et al., 28 Nov 2025).

1. Unsupervised Grouping of Children’s Speech

GRAPAM operates on a labeled corpus $D = \{(x_i, y_i)\}_{i=1}^N$ of speech utterances $x_i$ and transcriptions $y_i$. Each utterance $x_i$ is embedded into a $d$-dimensional vector $e_i \in \mathbb{R}^d$ to capture salient acoustic characteristics. The framework explores several embedding strategies:

• Speaker embedding: $e_i \in \mathbb{R}^{192}$ derived from a pre-trained ECAPA-TDNN.
• Low-level speech metrics (LSM): Features such as pitch, pause rate, and spectral centroid, reduced via PCA to $d=16$.
• Zero-shot WER-based scalar (ZS-WER): $d=1$, representing a proxy for utterance-level ASR difficulty.

The complete set of embeddings $E = \{e_i\}_{i=1}^N$ is partitioned by $K$-means clustering to yield $K$ groups $\{C_1,\ldots, C_K\}$, optimizing:

$$\min_{C_1,\ldots,C_K} \sum_{k=1}^K \sum_{e_i \in C_k} \| e_i - \mu_k \|_2^2, \quad \mu_k = \frac{1}{|C_k|} \sum_{e_i \in C_k} e_i$$

Each cluster $C_k$ maps to a data subset $D_k$, thus forming $D = \bigcup_{k=1}^K D_k$.

2. Group-Specific Partial Fine-Tuning

Starting from an adult-pretrained Whisper-medium model ($\theta_0$, approximately 763.9M parameters), GRAPAM introduces parameter-efficient fine-tuning by selectively adapting only specific submodules:

• $P_{\mathrm{FFN}}$: Feed-forward submodules across all Transformer layers.
• $P_{\mathrm{ATTN}}$: Attention submodules.

Parameters are partitioned into a frozen subset $\theta_0^{-}$ and an adaptable subset $\theta_0^{+}$, with $\theta_0^{+} \subset \theta_0$. For each group $k=1\ldots K$ (and the entire dataset, denoted “all”), model adaptation solves:

$$\theta_k = \mathsf{PFT}(\theta_0, D_k, P_{\mathrm{choice}})$$

with the loss:

$$\mathcal{L}(\theta; D_k) = -\sum_{(x, y) \in D_k} \log p_\theta(y|x) + \lambda \|\theta_0^+\|^2$$

where typically $\lambda = 0$. Only $\theta_0^{+}$ is updated by gradient descent, leveraging AdamW with batch size 16, learning rate $1 \times 10^{-5}$, and a single epoch. Fine-tuning is conducted for all combinations: all parameters (SFT), only FFN, and only ATTN.

3. Parameter-Level Model Merging

After group-wise fine-tuning, GRAPAM merges the resulting models using parameter-wise linear interpolation (LERP). Given a set of models indexed by $M \subseteq \{\text{"all"}, 1, \ldots, K\}$, the merged parameters are:

$$\theta_{\text{merge}} = \sum_{i\in M} \alpha_i \theta_i, \quad \sum_{i\in M} \alpha_i = 1,\, \alpha_i \geq 0$$

In reported experiments, uniform weights $\alpha_i = 1/|M|$ are used. The merged model, $\theta_{\text{merge}}$, incorporates both group-global and group-specialized adaptations, exploiting diversity across clusters to improve generalization.

4. Unified Optimization Framework

The entire pipeline can be viewed as a compositional optimization problem:

1. Partition the data by clustering acoustic embeddings:

$$\{C_1, \ldots, C_K\} = \arg\min_{C} \sum_{k=1}^K \sum_{e \in C_k} \|e - \mu_k\|^2$$

2. For each data group $i\in\{\text{"all"}, 1, \ldots, K\}$, conduct partial fine-tuning:

$$\theta_i = \arg\min_{\theta} \mathcal{L}(\theta; D_i), \qquad \theta^{-} \text{ frozen}$$

3. Merge the resulting models:

$$\theta_{\text{merge}} = \sum_{i\in M} \alpha_i \theta_i$$

The method’s hyperparameters thus comprise $K$ (number of clusters), $P_{\mathrm{choice}}$ (layer subset for adaptation), $\eta$ (learning rate), and $\{\alpha_i\}$ (merging weights).

5. Empirical Evaluation and Results

Experiments are performed on the MyST (My Science Tutor) children's speech corpus, filtered to 42,790 training utterances (117 hours, 566 speakers), 6,812 validation utterances (18 hours), and 7,257 test utterances (19 hours). The Whisper-medium model is configured with 24 encoder and 24 decoder Transformer layers.

Key settings include $K=3$ clusters (speaker-embeddings, LSM, ZS-WER, and random assignments as a control), and partial fine-tuning on:

• All parameters (SFT, 763.9M params)
• Only FFN (402.9M params)
• Only ATTN (302.4M params)

The primary metric is Word Error Rate (WER). Relative improvement is computed as:

$$\Delta\mathrm{WER}_{\mathrm{rel}} = 100\% \times \frac{\mathrm{WER}_{\mathrm{ref}} - \mathrm{WER}_{\mathrm{model}}}{\mathrm{WER}_{\mathrm{ref}}}$$

A summary of experimental results:

Method	WER (%)	Relative WER Improvement (%)
Baseline (no FT)	14.05	—
Full-model FT (all θ)	9.95	↓29.2
GRAPAM, full SFT merge	9.36	↓33.4
GRAPAM, PFT FFN merge	9.31	↓33.7
GRAPAM, PFT ATTN merge	9.38	↓33.2

Relative to full-model fine-tuning, GRAPAM with PFT FFN (speaker embedding clustering, merging “all”, 1, 2, 3) achieves the best configuration: a further 6% relative reduction in WER, despite updating approximately half the parameter count.

Averaging across cluster/fine-tuning combinations, WER using PFT FFN is approximately 9.47%. This suggests group-aware merging is robust to clustering modality, and consistently beneficial.

6. Implications and Context in Pediatric ASR

GRAPAM’s group-aware adaptation pipeline leverages the diversity of children’s speech by constructing models specialized for acoustically homogeneous subgroups and merging their expertise. Traditional supervised fine-tuning on the global dataset misses such group-level acoustic idiosyncrasies and thus underperforms relative to the merger of group-specialized partial adapters. The observed efficiency—in both parameter savings and performance—positions model merging as a scalable adaptation technique for diverse, data-limited speech domains. A plausible implication is that similar methods could generalize to other settings with high intra-domain variability and limited adaptation data (Rolland et al., 28 Nov 2025).