Extend VASPO to multi-turn dialogue

Extend Value-Augmented Self-Distillation Policy Optimization (VASPO), which currently trains single-turn generation models using privileged reference information for critic value estimation and self-teacher modulation, to multi-turn dialogue in which conversational history and available privileged evidence evolve across successive turns.

Background

VASPO is designed for single-turn generation: the actor generates a response from the standard prompt, while a privileged critic and self-teacher access fixed reference information during training. Multi-turn dialogue introduces changing conversational history and potentially changing privileged evidence across turns, requiring the method to determine how these evolving inputs should be incorporated into value estimation and teacher guidance.

The paper explicitly identifies application to this setting as unresolved future work rather than addressing it experimentally or theoretically.