SDPO for alignment in open-ended or continuous‑reward settings

Determine whether Self‑Distillation Policy Optimization (SDPO), which distills a feedback‑conditioned self‑teacher into the policy, improves alignment in open‑ended text generation and in continuous‑reward tasks that lack a ground‑truth verifier, by empirically evaluating its retrospection‑based credit assignment in such settings.

Background

The paper introduces Reinforcement Learning with Rich Feedback (RLRF) and proposes Self‑Distillation Policy Optimization (SDPO), which uses a feedback‑conditioned self‑teacher to provide dense, logit‑level credit assignment. Experiments primarily focus on verifiable domains such as code generation, where rich environment feedback (e.g., runtime errors, unit tests) is available.

The authors note that many real‑world tasks provide textual feedback without a ground‑truth verifier. Extending SDPO beyond verifiable settings raises the question of whether the retrospection mechanism can improve model alignment when rewards are not purely binary or verifiable, such as in open‑ended text generation or tasks with continuous rewards.

References

While we focused on verifiable code generation, many tasks provide textual feedback without a ground-truth verifier. Investigating whether SDPO's retrospection mechanism can improve alignment in open-ended text generation or continuous-reward tasks remains an open empirical question.

— Reinforcement Learning via Self-Distillation  (2601.20802 - Hübotter et al., 28 Jan 2026) in Conclusion, Limitations, and Future Work — Future Work (Beyond verifiable rewards)

This leaves open how to incorporate teacher evidence into group-relative comparison and localize task credit without altering response-level scale.

— When and Where to Trust the Teacher: Unifying On-Policy Distillation and GRPO through Entropy-Calibrated Credit Assignment  (2609.28385 - Zhang et al., 23 Sep 2026) in Section 1, Introduction

All benchmarks have automatically checkable answers, so performance on open-ended tasks remains untested.

— Learning from Self-Generated Responses: Reliability--Uncertainty Self-Distillation for Reasoning  (2610.19176 - Pan et al., 13 Oct 2026) in Limitations, “Cost and scope,” p. 9