Integrate vision-language models into Attune

Integrate vision-language models into Attune’s operator-reviewed visual-pattern pipeline to support gaze-shift annotation from visual context rather than only symbolic object labels and annotated robot activity.

Background

Attune currently uses LLMs to reason over structured metadata: YOLO-derived visual cues, annotated robot activities, gaze timing, and operator-reviewed patterns. The paper explicitly identifies vision-language-model integration as a future direction, which would allow the system to incorporate raw or richer visual context when generating gaze-shift annotations.

References

Future work should explore live or near-real-time annotation despite its potential cognitive cost.

— Attune: A Self-Annotation Tool for Understanding Robot Operator Attention Profiles  (2608.12650 - Zhou et al., 12 Aug 2026) in Section 5, Limitations and Future Work

Future work should examine how LLM accuracy affects calibration validity, verification effort, and trust.

— Attune: A Self-Annotation Tool for Understanding Robot Operator Attention Profiles  (2608.12650 - Zhou et al., 12 Aug 2026) in Section 5, Limitations and Future Work

Third, the study did not include task-based or visual-saliency baselines, so it cannot objectively distinguish task-driven from salience-driven shifts; future work should compare operator-endorsed rationales with such baselines.

— Attune: A Self-Annotation Tool for Understanding Robot Operator Attention Profiles  (2608.12650 - Zhou et al., 12 Aug 2026) in Section 5, Limitations and Future Work

VLM integration, conditioned on operator-reviewed visual patterns, is left for future work.

— Attune: A Self-Annotation Tool for Understanding Robot Operator Attention Profiles  (2608.12650 - Zhou et al., 12 Aug 2026) in Section 3, Implementation