Controllable Spoken Dialogue Generation: An LLM-Driven Grading System for K-12 Non-Native English Learners

Abstract: LLMs often fail to meet the pedagogical needs of K-12 English learners in non-native contexts due to a proficiency mismatch. To address this widespread challenge, we introduce a proficiency-aligned framework that adapts LLM outputs to learner abilities, using China's national curriculum (CSE) as a representative case. Our framework enables precise control over lexical complexity through a four-tier grading system, supported by a comprehensive suite of new resources: graded vocabulary lists and a multi-turn dialogue corpus. Our core technical contribution is the \textbf{DDPO} algorithm,Diversity Driven Policy Optimization, a multi-turn GRPO-based approach designed to preserve dialogue diversity while holistically optimizing dialogue quality. This method significantly outperforms conventional approaches, achieving low out-of-vocabulary rates and high diversity while enhancing conversational naturalness and pedagogical value. While grounded in the CSE, our framework is designed for flexibility and can be readily adapted to other educational standards. Our models, data, and code will all be open-sourced, providing a scalable platform for personalized English speaking practice that effectively addresses the unique challenges faced by K-12 learners in non-immersive environments.

• The paper introduces a novel DDPO RL framework that jointly enforces graded vocabulary constraints while preserving dialogue diversity.
• The approach employs paired teacher-student models with hard vocabulary masking to simulate authentic, pedagogically-aligned dialogues.
• Empirical evaluations demonstrate that DDPO achieves low out-of-vocabulary rates and superior dialogue quality compared to baseline and SFT models.

Controllable Dialogue Generation for K-12 Non-Native English Learning: Technical Perspectives and Implications

Problem Formulation and Motivations

The paper "Controllable Spoken Dialogue Generation: An LLM-Driven Grading System for K-12 Non-Native English Learners" (2604.22542) presents a rigorous approach to learner-aligned spoken dialogue generation. The authors identify a core challenge: state-of-the-art LLMs, when applied generically, generate dialogue that is often mismatched with the cognitive and linguistic profiles of K-12 learners in non-native English education environments. Specifically, conventional LLM-driven systems neglect fine-grained control over lexical and grammatical complexity tailored to standardized proficiency bands (such as the CSE, used in China), which is essential for scaffolding oral language development consistent with best pedagogical theory and practice.

Existing approaches—ranging from simple prompt engineering to hard-constrained decoding—fail to jointly optimize both strict proficiency adherence and the diversity/naturalness critical for engagement and uptake in language learning. Moreover, the lack of CSE-aligned open resources and benchmarks further impedes progress on this front.

Dataset Construction and Graded Resources

A central contribution is the curation of a multi-tier dataset grounded in the CSE framework, which dissects proficiency into four operational levels (L1–L4). The authors algorithmically extract and organize graded vocabulary lists directly from Ministry of Education-sanctioned corpus resources and mainstream textbooks. Topic domains and communicative scenarios are aligned with both curricular and assessment standards, ensuring that the generated dialogues are authentic in both form and topic distribution.

Dialogues are constructed using paired constrained LLMs: separate teacher and student models simulate authentic conversational exchanges, with hard vocabulary masking to enforce grade-level compliance. The authors further employ subsequent rewriting with advanced unconstrained models (e.g., deepseek-v3) to mitigate fluency and grammaticality deficits introduced by the constrained procedure. Expert screening eliminates residual out-of-vocabulary (OOV) errors and context-incongruent content, resulting in a graded, high-quality, multi-turn dialogue corpus.

Methodology: Diversity Driven Policy Optimization (DDPO)

The technical core of the system is DDPO (Diversity Driven Policy Optimization), a novel RL-based algorithm extending Group Relative Policy Optimization (GRPO). The method explicitly addresses entropy collapse—a homogenization of generated outputs induced by local maxima exploitation in the reward landscape, commonly observed in generative RL for dialogue tasks. Entropy collapse is particularly problematic in educational dialogue, as it yields repetitive, mechanistic utterances unsuited for engaging or effective teaching.

DDPO is built around multi-turn, group-based trajectory sampling. Given a static dialogue prompt, the model samples multiple independent rollouts, each consisting of multi-turn exchanges within a simulated dialogue environment. Rewards are dynamically composed from three axes: (1) vocabulary adherence (hard rule-based and/or reward model guidance), (2) single-turn diversity (penalizing token-level similarity across rollouts for the same context), and (3) multi-turn trajectory diversity (penalizing within-rollout repetitions). Reward component coefficients are dynamically adjusted based on training progression, balancing exploration and exploitation. The policy is updated using a PPO-variant objective with token-level and group-average normalization to avoid bias and stabilize RL training. This architecture produces policies that maintain strong fidelity to proficiency constraints while maximizing both inter- and intra-session utterance diversity.

Empirical Results and Analysis

The paper provides comprehensive empirical benchmarking across baseline (pretrained), SFT (supervised fine-tuned), and RL-optimized agents (GRPO, DDPO), using both Qwen2.5 and Llama3.1 backbone architectures. Three evaluation dimensions are prioritized: (1) OOV rate (strict adherence to graded vocabulary), (2) diversity score (Rouge-L based, inter/intra-session), and (3) pedagogical dialogue quality (multi-facet, including topic relevance, information richness, and topic guidance).

Strong numerical evidence establishes that:

• Prompting and unconstrained base models fail to enforce proficiency compliance, yielding OOV rates consistently above 40%.
• Constrained decoding eliminates most OOV errors but at the cost of uncontrolled grammatical/semantic errors and degenerate outputs.
• SFT yields moderate improvement in both OOV control and diversity/fluency, but cannot guarantee full alignment with graded constraints.
• DDPO achieves vocabulary violation rates competitive with hard constrained approaches ($\sim$6–9%), and simultaneously outperforms SFT and GRPO on diversity and multi-aspect dialogue quality metrics.

A detailed ablation shows that naïve constraint-optimization (GRPO) drives diversity to collapse (score $\sim$0.39), while full DDPO restores near base-model diversity (0.72) without sacrificing constraint adherence. Furthermore, the model-based reward signal is necessary for substantial gains in semantic richness and naturalness.

Model response case studies underline the impact of entropy collapse in GRPO (nearly deterministic, trivial outputs), while DDPO maintains diverse, contextually appropriate and proficiency-aligned instructional dialogues.

Practical and Theoretical Implications

On the practical side, the proposed framework demonstrates generalized efficacy across multiple modern LLM architectures. It provides a scalable template for constructing open-source, policy-controlled English dialogue systems that can be swiftly adapted to any standards-driven educational regime (e.g., CEFR, local adaptations). The public release of both graded corpora and trained policies removes a significant bottleneck for future research.

Theoretically, the work advances the field of controlled text generation by demonstrating that multi-objective RL optimization, when properly structured, can simultaneously resolve hard constraints and generative diversity. The explicit modeling and mitigation of entropy collapse in dialogue RL represents a methodological advancement with broad implications for controllable sequence modeling in other domains. The system also demonstrates that LLM-based evaluation can closely match expert human ratings for educational dialogue, enabling scalable benchmarking.

Limitations and Future Directions

Notable limitations include reliance on synthetic user simulation (rather than real student interactions), increased computational overhead from group-sampling in DDPO, and the static nature of curated vocabulary lists which may miss productive morphological variants. The framework has not yet been scaled to very large LLMs or to highly typologically diverse languages, and integration of multi-modal modalities remains untapped.

Future research is likely to pursue real-user-in-the-loop RL for increased ecological validity, dynamic or adaptive vocabulary policies, and multi-modal spoken dialogue with prosodic and pronunciation modeling. The DDPO architecture provides a modular foundation for such exploration.

Conclusion

This work presents a significant technical advancement in the design of proficiency-aligned spoken dialogue agents for K-12 non-native English education. By integrating curriculum-aligned graded resources with the DDPO RL framework, the system robustly enforces pedagogical constraint while preserving the generative diversity essential for effective spoken language practice. The released resources and models establish a reproducible and extensible benchmark for future research on controlled and adaptive educational dialogue agents.