Quantitatively evaluate the accuracy of the OnGoal goal pipeline

Develop and conduct a quantitative evaluation of the accuracy of OnGoal’s three-stage goal pipeline (infer, merge, evaluate) using expert-annotated benchmark datasets to validate goal identification, merging operations, and evaluation categories against ground truth labels.

Background

OnGoal employs a three-stage pipeline that infers, merges, and evaluates conversational goals during multi-turn dialogue with LLMs. The study focused on user-reported accuracy and usability impacts rather than formal pipeline validation.

A quantitative, benchmark-driven assessment would clarify reliability and generalizability of the pipeline, enabling more rigorous comparisons across models, prompts, and task contexts.

References

However, quantitatively evaluating our pipeline's accuracy, such as on expert-annotated benchmarks, remains untested.

— OnGoal: Tracking and Visualizing Conversational Goals in Multi-Turn Dialogue with Large Language Models  (2508.21061 - Coscia et al., 28 Aug 2025) in Section 7.2 Limitations and Future Work

Quantifying recall remains the main outstanding validation step, and awaits an independently annotated corpus of Circulars, which does not yet exist for this task.

— AI-Assisted Extraction of Follow-up Observations from GCN Circulars in Astro-COLIBRI  (2608.23270 - Schüssler et al., 24 Aug 2026) in Conclusion; see also Section 7.2, “Future work” (Section \ref{sec:future_work})