Evaluate requirement coverage in end-to-end generated dialogue trees

Evaluate whether end-to-end generated NPC dialogue trees produced from Knudge quest specifications satisfy all quest requirements and complete all user-provided specifications, thereby "checking all the boxes" of required objectives.

Background

The paper primarily evaluates next-utterance prediction and conducts a limited case study of full tree generation, acknowledging that these generated trees do not approach the complexity of actual in-game dialogue structures. While the dataset, Knudge, encapsulates granular quest objectives and lore constraints, the authors note that they have not yet performed a comprehensive evaluation of whether end-to-end generated dialogues meet all listed requirements.

Consequently, a systematic evaluation protocol to verify that generated dialogue trees fully cover and comply with the specified quest objectives and user-provided specifications remains to be developed, and the authors explicitly leave this task for future work.

References

We also leave for future work the evaluation of whether end-to-end generated dialogues 'check all the boxes' of quest requirements, completing all user-provided specifications.

— Ontologically Faithful Generation of Non-Player Character Dialogues  (2212.10618 - Weir et al., 2022) in Limitations

Residual failures in requirement tracking and evidence coverage remain open problems.

— Learning from Research: Toward Lifelong Agent Harness Evolution  (2609.40169 - Yang et al., 30 Sep 2026) in Section 5, Conclusion