Imitation Learning for Elder-Facing Speech Synthesis

Abstract: Recent advances in text-to-speech (TTS) synthesis have achieved highly natural and expressive speech generation. However, these systems are designed for general adults and overlook older adults' speech comprehension needs due to age-related sensory and cognitive decline. Prior work involves older adults by collecting preference feedback to tune model parameters. However, obtaining sufficient preference data is costly and difficult, as older adults quickly become fatigued during collection. In this paper, we propose a novel imitation learning (IL) framework to learn TTS models from expert demonstrations. We further improve Group Relative Policy Optimization (GRPO) with two-stage on-policy reward learning (OPRL) to mitigate reward hacking under limited supervision from expert demonstration. Experimental results show that GRPO w/ OPRL outperforms GRPO and supervised baselines in objective and subjective metrics. Audio samples are available at https://dongru1.github.io/demo/im-efss

• The paper leverages expert demonstrations and imitation learning to overcome the scarcity of elderly-specific speech data for TTS customization.
• It employs a two-stage On-Policy Reward Learning process that combines prosodic embeddings and syllable error rate metrics to mitigate reward hacking.
• Experimental results show that the OPRL-augmented GRPO reduces SER/CER and delivers higher MOS scores compared to conventional baselines.

Imitation Learning Framework for Elder-Facing Speech Synthesis

Motivation and Background

The demographic trend toward aging populations globally imposes increasing demands for accessible and intelligible speech synthesis, particularly tailored to the sensory and cognitive requirements of older adults. Conventional TTS systems are typically optimized for general adult listeners, overlooking age-specific factors such as elevated hearing thresholds and distinct prosodic preferences. Direct preference collection from elderly users is labor-intensive and practically constrained by rapid fatigue; prior efforts in human-in-the-loop optimization, for example "HILvoice: Human-in-the-Loop Style Selection for Elder-Facing Speech Synthesis" [chenHILvoiceHumanintheLoopStyle2022], suffer from scalability limitations.

To circumvent the limited availability of subjective preference data from older adults, the paper introduces a meta-optimization approach utilizing expert demonstrations recorded by healthcare professionals as proxies for optimal elderly-directed speech. This paradigm leverages inverse reinforcement learning (IRL) to infer intrinsic reward functions guiding TTS model adaptation based on observable expert behaviors.

Pipeline and Reward Modeling

The methodology is anchored in imitation learning, where expert demonstrations constitute the positive samples, and neutral/prototypical TTS output forms the negative set. The reward model employs a frozen prosodic encoder from StyleTTS 2 [liStyleTTS2Humanlevel2023] to extract 128-dimensional prosodic embeddings, which are subsequently processed by a multi-layer residual network reward head to produce an unbounded score normalized to $(0,1)$.

In addition to style preference (expert reward), the pipeline augments the intrinsic reward signal with a pronunciation reward based on syllable error rate (SER) obtained from a production-grade Cantonese ASR (SenseVoice-small) [anFunAudioLLMVoiceUnderstanding2024]. The composite reward is a weighted harmonic mean, thus imposing stringent constraints and preventing optimization shortcuts where prosodic alignment is maximized at the expense of intelligibility (reward hacking).

Figure 1: Schematic of GRPO pipeline integrating on-policy reward learning (OPRL) with expert demonstration-based reward modeling.

On-Policy Reward Learning and GRPO

The paper extends the Group Relative Policy Optimization (GRPO) protocol [liuGroupRelativePolicy2025] by introducing a dynamic, two-stage On-Policy Reward Learning (OPRL) procedure:

• Stage 1: Iteratively updates the reward model with generated rollouts from the TTS policy fine-tuned by GRPO, enriching the training set with rollouts scored between negative and expert samples and retraining the reward model. This maintains reward function relevance as the policy distribution evolves, mitigating reward hacking.
• Stage 2: Further generalizes the reward model using external text data, binning generated samples by SER and ranking by predicted expert reward. Quantile-based reward assignment enables learning from text domains absent in demonstration data by relying on relative quality rankings.

The GRPO policy objective is regularized with a KL term penalizing excessive deviation from the reference model, ensuring stability and preventing mode collapse.

Experimental Results

Quantitative evaluation demonstrates the effectiveness of OPRL-augmented GRPO. The baseline, CosyVoice2-Yue, undergoes supervised fine-tuning (SFT) on the expert demonstration set for initialization. GRPO w/o OPRL exhibits pronounced reward hacking, evidenced by excessive speech silences and duration metrics diverging from ground truth ($\text{Dur}_\text{sil}$: 11.51 vs. 5.43, $Dur$: 27.62 vs. 19.27), and is outperformed in objective metrics by SFT and GRPO w/ OPRL.

GRPO w/ OPRL Stage 1 reduces SER and CER and aligns style and intelligibility more closely with expert targets. Stage 2 further improves prosody and achieves highest MOS scores (best: 3.78), statistically significant versus all baselines ($p<0.01$ vs. CosyVoice2-Yue and GRPO w/o OPRL; $p<0.05$ vs. all baselines).

Figure 2: Mel-spectrograms and pitch contours: OPRL-based reward models distinguish ground-truth from generated speech, preventing reward gaming.

The OPRL-trained reward model assigns lower scores to policy-generated samples compared to naïve reward models. Visualization shows GRPO w/o OPRL generates speech with prolonged silences and unnatural pacing, while OPRL-based GRPO achieves prosodic alignment without sacrificing intelligibility.

Implications and Future Directions

This work validates the practical viability of imitation learning frameworks for low-resource preference alignment in specialized TTS domains where direct human labeling is infeasible. The two-stage OPRL process dynamically adapts rewards to evolving policy distributions, addressing reward hacking and maintaining alignment with complex, multi-objective criteria.

The approach outlines a scalable path toward TTS systems customized for population subgroups with non-standard comprehension profiles. Theoretically, it suggests that composite reward modeling and on-policy learning are critical for navigating the RLHF vulnerabilities in generative speech tasks.

Future research may extend this paradigm to include direct preference evaluation from older adults and dissect inter-group preference differences. More sophisticated hierarchical reward models, expanded expert demonstration datasets, and cross-lingual adaptation remain promising avenues for robust, generalizable preference-aligned synthesis.

Conclusion

The paper proposes an imitation learning strategy for elder-facing TTS that leverages expert demonstrations and on-policy reward optimization. The GRPO w/ OPRL pipeline achieves superior alignment with elderly preferences, significantly outperforming conventional SFT and RL-based baselines in both objective intelligibility metrics and subjective MOS ratings. The dynamic reward refinement process proves essential for mitigating reward hacking and maintaining multi-faceted speech quality. These findings highlight the necessity of tailored RLHF protocols for adaptive speech synthesis in aging societies, with broad applications for accessibility technology and personalized interaction agents.