GuideEval: Evaluating AI Guidance

• GuideEval is a comprehensive framework that systematically evaluates AI guidance and intent understanding across various interactive systems.
• It employs diverse methodologies including behavior state detection, intent prediction, and dual-path heuristic evaluations to measure user assistance in GUI tasks, visual analytics, and educational dialogues.
• These evaluations highlight current limitations in multimodal and language models, driving research towards more context-sensitive and adaptive AI guidance.

GuideEval

GuideEval is a family of methodologies and benchmarks for systematically evaluating guidance and intent understanding in artificial intelligence systems interacting with humans. While the term "GuideEval" appears in diverse contexts—ranging from GUI intent assistance to instructional dialog and guidance in visual analytics—the unifying principle is the rigorous measurement of how well AI agents support, guide, or adapt to user needs, intentions, and cognitive states. These frameworks provide formal protocols, annotated datasets, and analytic metrics for both automated and human-in-the-loop assessment.

1. Benchmarking User Understanding and Assistance in GUI Tasks

GuideEval, as formalized in GUIDE: GUI User Intent Detection Evaluation, addresses the evaluation of AI models' ability to assist users in open-ended GUI workflows by moving beyond mechanical click automation toward collaborative, context-sensitive support (Yang et al., 26 Mar 2026). It operationalizes the evaluation of GUI agents through three core tasks:

1. Behavior State Detection: Classifies the user's behavioral state (nine classes—e.g., Exploration & Decision-Making, Debugging, Frustration) from screen recordings without access to audio.
2. Intent Prediction: Infers the user's immediate goal, presented as a four-way multiple-choice question (e.g., “add a progress bar”).
3. Help Prediction: Determines whether the user needs assistance and, if so, which type of help best fits the current context.

The dataset comprises 67.5 hours of annotated screen recording sessions from novice users across ten creative and analytical applications, with high-fidelity "think-aloud" narrations and both automated (WhisperX, Gemini-2.5-Pro) and human-refined annotations. A vision-only, zero-shot evaluation protocol is mandated, using 32 sampled frames per segment and optional context history.

Performance is measured via accuracy for classification tasks and Multi-Binary Accuracy (MBAcc) for multi-choice settings:

$$\text{Accuracy} = \frac{1}{N}\sum_{i=1}^N\mathbb{I}(\hat y_i=y_i)$$

$$\text{MBAcc} = \frac1N\sum_{i=1}^N \prod_{c\in\mathcal C_i^-}\mathbb{I}(f(x_i,y_i,c)=y_i)$$

Help need detection uses standard binary classification metrics (precision, recall, F1).

GuideEval has demonstrated that state-of-the-art MLLMs achieve only moderate performance (e.g., 44.61% accuracy on behavior state detection, 71.39% on intent prediction, and 55.00% on help content), but injecting structured context dramatically boosts scores (e.g., up to +50.2 percentile points for help prediction), highlighting the criticality of user modeling and temporal context handling in GUI assistants (Yang et al., 26 Mar 2026).

2. Dual-Path Heuristic Evaluation in Visual Analytics

GuideEval in the context of visual analytics, introduced by Ceneda et al., provides a comprehensive, dual-path methodology for quantifying guidance effectiveness (Ceneda et al., 2023). It formalizes eight Quality Criteria, each mapping to a distinct dimension of effective guidance:

• Flexible adaptation to context and user needs
• Adaptive personalization of content
• Visibility and transparency in guidance cues
• User control over guidance
• Explainability of system suggestions
• Expressive and interpretable communication
• Timeliness of delivery
• Relevance to analytic goals

For systematic evaluation, GuideEval supplies parallel expert- and user-oriented heuristic rubrics (18- and 27-item respectively), with each item scored on a 1–7 Likert scale. Expert evaluations target prototype and design-stage aspects; user evaluations focus on experience and practical task completion. Scores are aggregated per-criterion and overall:

$$\text{Criterion}_k = \sum_{h\in H_k} r_h \quad , \quad \text{Overall} = \sum_{k=1}^8 \text{Criterion}_k$$

This dual evaluation helps expose design omissions and runtime practicalities, providing a nuanced diagnostic profile guiding tool improvement.

3. Adaptive Guidance in Educational Dialog with Socratic LLMs

Within the domain of instructional dialog, GuideEval provides a behavioral evaluation framework for Socratic LLMs, emphasizing adaptive pedagogical strategies rather than static content delivery (Liu et al., 8 Aug 2025). The framework decomposes tutoring into three behavioral phases:

• Perception: Inferring discrete learner states (Accurate, Erroneous, Comprehension, Confusion) from student utterances.
• Orchestration: Deploying instructional strategies (Advance or Reconfigure) conditional on learner state, e.g., advancing when appropriate or scaffolding under confusion.
• Elicitation: Stimulating reflection via depth-calibrated questions mapped to Bloom’s taxonomy.

Empirical evaluation leverages a balanced corpus of real and synthetically perturbed student–tutor interactions, assessing model behavior using manual and LLM-applied rubrics (e.g., P-Affirm, P-Redirect, E-Heuristic):

Metric	Definition	Score Range
P-Affirm	Explicit affirmation of correct answers	0, 0.5, 1
P-Redirect	Correction of errors/misconceptions	0, 1
O-Advance	Whether the tutor meaningfully progresses instruction	0, 1
O-Reconfigure	Adaptive restructuring toward error/confusion	0, 1
E-Strategic	Use of higher-order questioning under correct comprehension	0, 1
E-Heuristic	Prompts for heuristic/discovery-based learning under confusion	0, 2, 3

Adaptivity indices (OSA and ESA) further quantify sensitivity to context:

$$\text{OSA} = \frac{1}{N}\sum_{i=1}^N \mathbb{I}(\text{strategy adapts across states})$$

$$\text{ESA} = \text{mean question depth change across contrastive states}$$

Existing LLMs typically underperform on seamless corrective feedback and adaptive orchestration, with fine-tuning on behavior-guided dialogs producing substantial gains in these metrics (Liu et al., 8 Aug 2025).

4. Diagnostic and Interpretable Evaluation in GUI Agent Trajectories

A further extension, focusing on the evaluation of GUI agents' task execution, employs hierarchical analysis to achieve both interpretability and long-horizon robustness (Zhai et al., 6 Apr 2026). The evaluation pipeline, termed GUIDE (GUI Understanding and Interpretable Diagnostic Evaluation), consists of:

1. Trajectory Segmentation: Partitioning long action–observation sequences into semantically coherent subtasks, leveraging LLM-prompted structure extraction.
2. Subtask Diagnosis: Assigning to each segment a verdict (success, partial, fail), diagnosing root causes, and issuing step-level corrective recommendations.
3. Overall Summary: Aggregating per-segment diagnoses into a task-level outcome with justification.

This modular decomposition drastically mitigates context overload and enables precise error localization. The system, evaluated on datasets spanning industrial e-commerce, web manipulation, and Android device control, achieves substantial gains in accuracy (up to +5.35 pp over prior evaluators) and maintains stability on trajectories as long as 80 steps. All modules operate zero-shot using backbone MLLMs with no further gradient training, relying on carefully engineered prompts and structured schema (Zhai et al., 6 Apr 2026).

5. Broader Impact, Limitations, and Future Directions

GuideEval frameworks consistently reveal a gap between the superficial interpretive ability of current multimodal and LLMs and the deeper, interaction-sensitive guidance required for effective human–AI collaboration. Key limitations identified include:

• Poor discrimination of nuanced user behavioral states, especially under trial-and-error or frustration, due to overgeneralization toward “Performing Actions” (Yang et al., 26 Mar 2026).
• Overgeneration and trivial guidance on web UIs when grounding is ambiguous or context is insufficient (Gan et al., 2 Feb 2026).
• Limited adaptivity in instructional dialogs, with LLMs frequently affirming erroneous responses or failing to recalibrate questioning strategy (Liu et al., 8 Aug 2025).
• The need for domain-specific priors and richer temporal memory for robust segmentation and diagnosis in trajectory-based evaluation (Zhai et al., 6 Apr 2026).
• Imperfect alignment of automatic text similarity metrics (BLEU, ROUGE) with end-user or domain-expert judgments of guidance value, necessitating exploration of alternative or learned metrics (Ceneda et al., 2023).

Recommended future directions include the development of stratified memory architectures for richer context embedding, integration of reward models and fine-grained scores for RL settings, expansion to multi-step/multi-page guide scenarios, and incorporation of user feedback loops for dynamic guide adaptation.

GuideEval thus establishes methodological baselines and experimental evidence for the field, mandating a shift toward context-rich, interpretable, and human-aligned guidance assessment across a spectrum of AI–human interface modalities.