Evaluate MASP with real-user interactions

Evaluate whether MASP’s mental-state annotations, rationality signal, and resulting policies perform reliably in interactions with legally and ethically recruited real users.

Background

The paper audits the BDI annotations and rationality judge against human judgments on sampled dialogue data, but these audits do not constitute evaluation of the dialogue policy with actual users. The authors therefore identify real-user evaluation as an unresolved direction.

References

These results support the consistency of the synthetic components on the audited samples, but they do not establish performance with real users. Evaluation with legally and ethically collected real-user interactions remains future work.

— Stop Training Against Soft Users: Mentalized Adversarial Self-Play for Proactive Dialogue  (2610.21156 - Liu et al., 15 Oct 2026) in Appendix, Section “Human Validation of Synthetic Components”